ChristianTremblay / pyhaystack / 177
40%

DEFAULT BRANCH: master
Ran
Jobs 4
Files 51
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
177

push

travis-ci

web-flow 
Release/0.92 (#49)

* First implementation of a browser that will help navigate the haystack server
from pyhaystack.client.niagara import NiagaraHaystackSession
from pyhaystack.browser.browser import Browser
import logging
logging.root.setLevel(logging.DEBUG)

session = NiagaraHaystackSession(uri='http://ip', username='user', password='password')
server = Browser(session)
site = server['Site'] will return the site
site['equipment'] will return an equipment

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* Equipments and points are created

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* New approach for high level interface
Adding functions to mixins

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* Adding helper functions to high level interface (__iter__, __getitem__, etc...)

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* client.http.base: Send Content-Length as a string.

`requests` tries to do a regex match or similar on the values of
headers, and this requires the input to be a string.  It barfs when you
give it an `int`.

* client.ops.vendor.widesky: Fix typo in create entity operation.

* client.ops.entity: Add loggers, handle empty grid.

* client.ops.his: Handle write of empty series.

* client.ops.his: Handle no columns in write_his_frame.

* WC-241: revive multi-hist mixin.

* WC-241: initialize timestamp column for write grid

* WC-241: Added a simple application that uses the pyhaystack library

Edit by Stuart Longland <stuartl@vrt.com.au>: Removal of default
credentials!

* WC-237: Re-locate manual mutli-his-read-write test script.

Manually run test scripts can go under tests/manual, until such time as
they get automated.

* WC-237: client.session: Add storage for grid cache.

Some grids are unlikely to change in the course of a session.  Examples
of these are the 'about', 'formats' and 'ops' grids, which indicate the
features that the Haystack server supports.  We should cache these for a
limited period of time so that the results can be used for enabling
features depending on server version.

* WC-237: client.ops.grid: Implement cache support

* WC-237: client.session: Enable caching for about, ops, formats.

* WC-237: client.mixins: Remove defunct classes from __init__

* WC-237: client.session: Fix typos

* WC-237: client.session: Add "has_feature" operation.

Used to detect extension features in the Haystack server.  Some servers
implement extension featurs that the standard Haystack API makes
difficult to expose.

Sometimes a feature is a "new" feature in a particular release of a
server, as such, it should be disabled if the version is found to be
below a certain threshold.  (e.g. multi-point hisRead/hisWrite is
supported in WideSky v0.5.0 but not earlier)

This allows a run-time check to take place in the relevant state
machines to decide how to tackle the problem.

* WC-237: client.ops.feature: Add HasFeatureOperation base implementation.

* WC-237: client.ops.feature: Fix some state machine bugs, typos.

* WC-237: client.session: Use base class for HasFeaturesOperation

* WC-237: client.ops.vendor.widesky: Add detection of WideSky extensions

* WC-237: client.widesky: Enable detection of WideSky extensions

* WC-237: client.ops.his: Probe multi-point hisRead

* WC-237: client.ops.his: Implement probing of multi-hisWrite

* WC-237: Pull in semver for WideSky version checking

* WC-237: client.ops.his: Fix multi-point hisRead/hisWrite issues.

* client.http.dummy: Dummy HTTP Client/Server

This is a "HTTP Server" class and matching "Client" that we can drive
like a puppet for writing test cases.

* tests: Bring test case for 'about' method.

* client.http.base: Handle 'str' input on Python3.

If we're given an 'str', we might safely assume that the body is in fact
decoded already.

* tests.util: Add some utility functions

* tests.client.test_base: Check output of about()

* tests.client.test_base: Add ops test

* tests.client.test_base: Add formats() test

* tests.client.test_base: More tests for read by ID, etc.

* getitem will now return tags or equip (for site) or points (for equip)

Some bugs are still there but WIP goes well

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* Corrected point mixin so value works
Added his shortcut to his_read_series (with a default to 'today' range
ex. znt.his(rng='yesterday') or znt.his()

filter requests works with __getitem__
ex. zone_sensors = my_office['sensor and zone']

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Bugfix: Handle missing entities in multi-entity read requests (#26)

* WC-366: client.ops.entity: Handle missing entities in read response.

If an entity does not exist, and we are requesting more than one, the
missing entity appears in the grid as a row of nulls:

    DEBUG:pyhaystack.client.WideskyHaystackSession.get_entity:Received grid: <Grid>
        Columns:
            id
            dis
            status
            user
        Row    0: id=Ref(u'test_user4', u'test_user4', True), dis=u'Test User4', status=u'inactive', user=MARKER
        Row    1: id=None, dis=None, status=None, user=None
        Row    2: id=Ref(u'test_user2', u'test_user2', True), dis=u'Test User2', status=u'inactive', user=MARKER
        Row    3: id=Ref(u'test_user3', u'test_user3', True), dis=u'Test User3', status=u'inactive', user=MARKER
    </Grid>

which then results in this exception:

      File "pyhaystack/client/ops/entity.py", line 51, in _on_read
        entity_id = row.pop('id').name  # Should be a Ref
    AttributeError: 'NoneType' object has no attribute 'name'

We should skip rows that have a null 'id'.

* WC-366: client entity tests: Test retrieval of entities by id.

* WC-366: client entity tests: Python 3 support

* client.ops.grid: Handle expired cache (#35)

If the cache has expired, then we can wind up with the situation where
`op` is `None`, so the `if` block falls through to the `else` branch
erroniously.

This causes a stack trace like the following:

```
    2017-02-17 01:49:00,039 pyhaystack.client.WideskyHaystackSession.his_write_frame[his.py: 726] WARNING Unable to probe multi-his-write support
    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 723, in _on_probe_multi
        result = operation.result
      File "/usr/lib/python2.7/dist-packages/pyhaystack/util/state.py", line 97, in result
        self._result.reraise()
      File "/usr/lib/python2.7/dist-packages/pyhaystack/util/asyncexc.py", line 28, in reraise
        reraise(*self._exc_info)
      File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/feature.py", line 93, in _on_got_about
        self._about = operation.result
      File "/usr/lib/python2.7/dist-packages/pyhaystack/util/state.py", line 97, in result
        self._result.reraise()
      File "/usr/lib/python2.7/dist-packages/pyhaystack/util/asyncexc.py", line 28, in reraise
        reraise(*self._exc_info)
      File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/grid.py", line 118, in _check_auth
        self._state_machine.auth_ok()
      File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 297, in fn
        self.transition()
      File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 290, in _tran
        self._enter_state(e)
      File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 341, in _enter_state
        return getattr(self, fnname)(e)
      File "/usr/lib/python2.7/dist-packages/fysom/__init__.py", line 88, in _callback
        return func(obj, *args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/grid.py", line 177, in _do_check_cache
        op.done_sig.connect(_proxy)
    AttributeError: 'NoneType' object has no attribute 'done_sig'
```

We need to instead do our check while we have the mutex locked and
disregard the expired cache.

* skyspark V3 SCRAM authentication (#36)

First Raw of Skyspark V3 SCRAM authentication

* Proposed updates for release 0.92 (#41)

* WC-414: Ensure HTTP header names are US-ASCII encoded.

python-requests does not like it when it sees Unicode here.

* WC-414: client.ops.grid: Ensure grid is encoded

For now, assume UTF-8 encoding since that seems to be the standard being
used elsewhere.

* WC-414: client.ops.vendor.widesky: Fix encoding issues.

Ensure all headers are encoded using US-ASCII encoding.
Ensure all body data is encoded using UTF-8 encoding.

We'll treat passwords as being UTF-8.

* WC-414: client.widesky: Send authorization in US-ASCII encoding.

The authorization header is basically a hash, so US-ASCII is the most
appropriate for this.

* WC-414: client.widesky: Assume already 'unicode' if no 'decode'.

If a string object has no 'decode' method, then assume it is a Unicode
object already.

* WC-414: client.http.base: Use bytes object not str/unicode

* WC-414: client.ops.grid: Use bytes object not str/unicode

* WC-414: base client tests: Fix encoding issues

- Headers should be bytes() objects
- Bodies should be decoded from UTF-8

* WC-414: client entity tests: Fix more encoding issues.

* FIR-214: client.http.base: Assert that we have a content type.

* FIR-214: client.http.base: Add accept_status argument.

This gives us the ability to whitelist specific status codes, instead of
failing normally "bad" codes such as 400, 401, 403 or 404, allowing the
underlying client code to handle it.

* FIR-214: client.http.dummy: Add accept_status logic.

* FIR-214: client.http.sync: Add accept_status logic.

* FIR-214: client.ops.grid: Pass through accept_status

* FIR-214: client.ops.grid: Add hook for detecting session issues.

Firefly will report error 401 if the session is invalid.

* FIR-214: client.widesky: Add hook to detect 401 error.

If we have a 401 error message thrown at us, assume we've been logged
out.  This could be delivered by way of a response or an exception.

We should not raise an exception ourselves, but instead just invalidate
the session.

* FIR-214: client: _on_http_response → _on_http_grid_response.

To imply that it's not just *any* HTTP response this gets called on, but
rather specific responses.

* FIR-214: client.widesky tests: Add test for is_logged_in.

* FIR-214: client.widesky tests: Test 401 drops the session.

* FIR-214: client.session tests: Check _on_http_grid_response gets called.

Add a test that verifies that the function *does* get called when a grid
is requested.

* FIR-214: client.widesky tests: Test other exceptions are ignored.

* client.ops.his: Don't define MetaSeries/MetaDataFrame unless pandas loaded

Otherwise you get stack traces like this when pandas isn't installed:

error	06-Sep-2016 14:45:22	======================================================================
error	06-Sep-2016 14:45:22	ERROR: Attempt to add a single user
error	06-Sep-2016 14:45:22	----------------------------------------------------------------------
error	06-Sep-2016 14:45:22	Traceback (most recent call last):
error	06-Sep-2016 14:45:22	  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
error	06-Sep-2016 14:45:22	    self.test(*self.arg)
error	06-Sep-2016 14:45:22	  File "/mnt/src/tests/test_user_editor_add.py", line 58, in test_add_single_user
error	06-Sep-2016 14:45:22	    core.main(arguments)
error	06-Sep-2016 14:45:22	  File "/mnt/src/widesky_editor/user/core.py", line 35, in main
error	06-Sep-2016 14:45:22	    session = init_haystack(args)
error	06-Sep-2016 14:45:22	  File "/mnt/src/widesky_editor/common.py", line 67, in init_haystack
error	06-Sep-2016 14:45:22	    http_args=http_args, tagging_model=WideSkyTaggingModel)
error	06-Sep-2016 14:45:22	  File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 61, in get_instance
error	06-Sep-2016 14:45:22	    implementation = get_implementation(implementation)
error	06-Sep-2016 14:45:22	  File "/usr/lib/python2.7/dist-packages/pyhaystack/client/loader.py", line 34, in get_implementation
error	06-Sep-2016 14:45:22	    package='pyhaystack.client')
error	06-Sep-2016 14:45:22	  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
error	06-Sep-2016 14:45:22	    __import__(name)
error	06-Sep-2016 14:45:22	  File "/usr/lib/python2.7/dist-packages/pyhaystack/client/widesky.py", line 8, in <module>
error	06-Sep-2016 14:45:22	    from .session import HaystackSession
error	06-Sep-2016 14:45:22	  File "/usr/lib/python2.7/dist-packages/pyhaystack/client/session.py", line 18, in <module>
error	06-Sep-2016 14:45:22	    from .ops import his as his_ops
error	06-Sep-2016 14:45:22	  File "/usr/lib/python2.7/dist-packages/pyhaystack/client/ops/his.py", line 801, in <module>
error	06-Sep-2016 14:45:22	    class MetaSeries(Series):
error	06-Sep-2016 14:45:22	NameError: name 'Series' is not defined

* WC-9: client.ops.vendor.widesky: Version detection tweaks

- Handle non-semver version string
- Prefer moduleVersion over productVersion; newer versions will use this
  over productVersion which will be the "WideSky release".  (like
  Ubuntu's; 16.04.)

* VRT-506: client.http.base: Make headers and cookies case-insensitive.

* VRT-506: client.http.base: Fix `bytes`/`str` issue, initialisation.

* client.loader: Add support for aliases for known implementations. (#44)

This allows us to whistle up aliases for implementations such as `ax`
for Niagara AX, etc.  Further implementations can be added to this file,
and they will be loaded when needed.

* Added support for easier session declaration using get_instance
we can now use :
```
import pyhaystack
pyhaystack.connect('ax', 'uri', 'username', 'password')

#or

import pyhaystack
pyhaystack.connect('ax', uri='uri', username='username', password='password')
```

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Feature/scram (#45)

* Just beginning to implement SCRAM script by Pierre into pyhaystack...
Modified state machine
and begun tests....
It is not working yet.

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Big step ahead. Now I get to the point where the Auth Failed message appears
The state machine seems ok

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Able to retrieve authToken (#37)

Working version retrieving the authToken, not able to send request
    afterwards, got response auth_attempt

* Connection is working now after modifying minor things in session.py
api=True by default
removed cookie from the response handling

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Clean up conversation comment and print statements

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Started work implementing Niagara4 SCRAM

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* WIP
I will need to access from another PC than the one running the Niagara 4 server...

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* file omitted

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* file omitted

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* WIP

* Just beginning to implement SCRAM script by Pierre into pyhaystack...
Modified state machine
and begun tests....
It is not working yet.

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Big step ahead. Now I get to the point where the Auth Failed message appears
The state machine seems ok

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Connection is working now after modifying minor things in session.py
api=True by default
removed cookie from the response handling

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Able to retrieve authToken (#37)

Working version retrieving the authToken, not able to send request
    afterwards, got response auth_attempt

* Clean up conversation comment and print statements

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Started work implementing Niagara4 SCRAM

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* WIP
I will need to access from another PC than the one running the Niagara 4 server...

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* file omitted

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* file omitted

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* WIP

* need rebase to benefit from headers lower case

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Added note folder to ignore

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* We receive server first message
Seems to fail with authentication
We are close !

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* final msg response is null.... why ?

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Fighting with session cookies

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* modified how I set cookies... no improvements

* Update niagaraV4.py (#43)

Working version Niagara, able to have the remoteSignature checked with the server one.
Will need to test with an other url afterwards to see if we are able to use the session to retrieve the data

* Modifications to salt password
added some print statements

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Removed manual handling of cookies and let the session plays...

* Niagara4 working
We needed one more request to j_security_check, no parameter to authenticate.

Now it's time to refactor, document and release !!!!

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Small changes before push.... this file will be deleted

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Cleanup, some comments.

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Deleted scripts that helped implementing scram

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Dropping Python 3.3 support

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Pierre Sigwalt Special Commit Edition

Added Pierre to the contributors file. He helped us with the implementation of SCRAM authentication for Skyspark and Niagara4.
Nice work that was way over my skills.
Thanks Pierre

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* First implementation of a browser that will help navigate the haystack server
from pyhaystack.client.niagara import NiagaraHaystackSession
from pyhaystack.browser.browser import Browser
import logging
logging.root.setLevel(logging.DEBUG)

session = NiagaraHaystackSession(uri='http://ip', username='user', password='password')
server = Browser(session)
site = server['Site'] will return the site
site['equipment'] will return an equipment

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* Equipments and points are created

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* New approach for high level interface
Adding functions to mixins

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* Adding helper functions to high level interface (__iter__, __getitem__, etc...)

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* getitem will now return tags or equip (for site) or points (for equip)

Some bugs are still there but WIP goes well

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* Corrected point mixin so value works
Added his shortcut to his_read_series (with a default to 'today' range
ex. znt.his(rng='yesterday') or znt.his()

filter requests works with __getitem__
ex. zone_sensors = my_office['sensor and zone']

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Small doc update
Should be able to get site from session in a more direct way

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Removed browser files as this is done using Mixin interfaces

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Added a notice when connection fails in the first place, to use a certificate or tls_verify = False

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Removed old folder

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Clean up on high level functions code
Merged with develop
Added session.sites and session.site, shortcut to retrieve the site from a server

Typical use :
```
import pyhaystack
session = pyhaystack.connect('n4', uri, username, password, http_args={"tls_verify":False})
my_site = session.site
my_site['UTA-1']['SF-O']

# UTA-1 is an equipment
# SF-O is a point under UTA-1

my_site['UTA-1']['SF-O'].his().result
# Will bring histories for this point

```

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Fix PBKDF2 ImportError on Debian Wheezy (#46)

* Clean up whitespace changes.

- Remove carriage returns
- Remove trailing whitespace

* client.ops.grid: Drop `print`

We should *not* be printing stuff to `stdout` for several reasons:

1. who says the user is watching `stdout`?
2. who says the user isn't using `stdout` for something else?

Randomly printing stuff has the potential to corrupt `stdout` output of
any script that calls the grid operations, which is pretty much all
scripts.  We should not be polluting `stdout` with this data.

* util.scram: Add hashbang, encoding to file.

* util.scram: Trim trailing blank lines.

* util.scram: Split excessively long lines.

* util.scram: Use backports.pbkdf2 as fallback.

If we can't load `pbkdf2` from `hashlib`, use the backports version.

* Disable SubjectAltNameWarning
Added more precise exception handling for SSLError in sync.py
This modification allow us to remove print statement in niagara_scram.py

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Modified README.rst to add new implementations (Niagara4 and Skyspark 3)

* replaced main README.rst by new one

* Error in nav arguments.....must be navId, not nav_id

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Update to README.rst to remove "new" implementation as the implementation is now established.

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* Added documentation around the square bracket search for site, equip, points.
Some cleanup probably needed.

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* docs: Clean up line endings, whitespace, etc.

* docs: connect.rst: More detailed connection information.

Detail the various options and their meanings.  Most options are
presented here for completeness.

* docs: connect.rst: Use `pyhaystack.connect`

This is an alias for `get_instance`… but Sphinx doesn't pick up on that
fact, and so I mention both there.

* docs: connect.rst: Move "from file" hint to Usage hints.

* docs: site.rst: Describe operations, high-level interface.

* Updated version
ready to push

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

* docs: tags.rst: Document tag CRUD operations.

Some servers, notably WideSky, support making changes to the asset model
through the API, and this is supported by pyhaystack.  This details how
to retrieve and how to update tags on an entity.

* docs: site.rst: Fix formatting, tweak wording

* docs: his.rst: Document timezones, range handling, etc.

* docs: site.rst: Fix reference to `TzMixin`

* docs: site.rst: Fix references, formatting

* docs: site.rst: Further formatting tweaks

* docs: connect.rst: Fix formatting

* docs: tags.rst: Formatting fixes, example of `is_dirty`

* docs: tags.rst: Fix broken references

* pyhaystack.info: Drop out middle initial.

Usually I just go by first/last name, the world doesn't need to know my
middle name is James. :-)

* docs: tags.rst: Fix more formatting issues

Keep forgetting that newline between the `::` and the start of the
block.

* More details on ways to deal with CA certificates when connecting to a server

Signed-off-by: Christian Tremblay, ing. <christian.tremblay@servisys.com>

* Minor corrections to comments into code
Removed references to LGPL and replaced by Apache 2.0
Renamed Stuart in contributors.txt

Signed-off-by: Christian Tremblay <christian.tremblay@servisys.com>

1065 of 2684 relevant lines covered (39.68%)

1.57 hits per line

Jobs
ID Job ID Ran Files Coverage
1 177.1 51
39.57
 Travis Job 177.1
2 177.2 51
39.23
 Travis Job 177.2
3 177.3 51
39.23
 Travis Job 177.3
4 177.4 51
39.23
 Travis Job 177.4
Source Files on build 177
Coverage ∆ File Lines Relevant Covered Missed Hits/Line