Version 2.0.07 - Token Refresh

Release Notes

Enhancements

• Added periodic token renewal functionality:
  • Implemented renew_token and periodic_token_renewal methods in SenseCollector to handle automatic token renewal at every specified interval.
    The token renewal interval is set to 12 hours (43200 seconds) by default, but it can be customized via environment variables.
  • Logging added to track token renewal actions and intervals.

Bug Fixes

• Fixed issue with environment variable comparison:
  • Corrected logic for case-insensitive comparison of environment variables to ensure proper detection of custom values.
  • Resolved issue where token renewal interval (SENSE_COLLECTOR_API_TOKEN_RENEW) was obscured incorrectly.

Logging Improvements

• Improved logging in main function:
  • Added debug logs for each step in the main function to provide detailed tracing of the initialization process.
  • Enhanced logging messages for authentication and environment variable retrieval.
  • Added logging for individual tasks within the asyncio.gather call to ensure visibility into task startup.

Docker and Build Script Adjustments

• Updated Dockerfile for new directory structure:
  • Moved Dockerfile to a parent directory and adjusted paths to copy necessary files from the src directory properly.
• Refined build script:
  • Transitioned build script from bash to Python for better readability and error handling.
  • Ensured real-time output during the build and push commands using subprocess.Popen for command execution.

New Environment Variables

• SENSE_COLLECTOR_API_TOKEN_RENEW:
  • Interval in seconds for token renewal. Default set to 43200 seconds (12 hours).

General Improvements

• Improved error handling and robustness:
  • Added error handling for all subprocess calls within the build script to ensure proper reporting and exit on failure.
  • Enhanced visibility into Docker build and push processes by merging stdout and stderr streams and ensuring immediate output flushing.

Documentation Updates

• Updated README and examples:
  • Examples of new environment variables and their usage were included.
  • Added documentation for new token renewal functionality and configuration.

v2.0.06 - Reconnection Logic Added

New Features and Enhancements

1. Device Data Caching and Concurrency Control:

   • Added Caching for Device Data: We introduced caching for device data with a configurable expiry time to reduce redundant API calls and improve performance.
     • Environment Variable: SENSE_COLLECTOR_DEVICE_CACHE_EXPIRY_SECONDS
     • Default Value: 120 seconds
   • Concurrent Device Lookups: This enhancement allows users to perform multiple device data lookups concurrently, up to a specified number of concurrent operations.
     • Environment Variable: SENSE_COLLECTOR_DEVICE_MAX_CONCURRENT_LOOKUPS
     • Default Value: 4
   • Lookup Delay Between Requests: To manage API rate limits effectively, we introduced a delay between consecutive device lookup requests.
     • Environment Variable: SENSE_COLLECTOR_DEVICE_LOOKUP_DELAY_SECONDS
     • Default Value: 0.5 seconds

2. Session Management:

   • Session Lifecycle Management: We have added methods to manage the lifecycle of aiohttp sessions, ensuring that sessions are properly opened and closed, which improves resource management and stability.

3. Always-On Functionality:

   • WebSocket Reconnection Logic: Implemented robust reconnection logic to maintain an "always-on" connection with the Sense API WebSocket feed. The application will now attempt to reconnect indefinitely with configurable intervals and backoff strategies.
     • Heartbeat Interval: A configurable interval sends heartbeat messages to keep the WebSocket connection alive.
       • Environment Variable: SENSE_COLLECTOR_WS_HEARTBEAT_INTERVAL
       • Default Value: 10 seconds
     • Heartbeat Timeout: Configurable timeout to wait for a response before considering the connection lost and attempting to reconnect.
       • Environment Variable: SENSE_COLLECTOR_WS_HEARTBEAT_TIMEOUT
       • Default Value: 30 seconds
     • Reconnection Strategy: Configurable initial delay, cap, and backoff factor for reconnection attempts.
       • Environment Variables:
         • SENSE_COLLECTOR_WS_RECONNECT_DELAY_INITIAL (Default: 5 seconds)
         • SENSE_COLLECTOR_WS_RECONNECT_DELAY_CAP (Default: 60 seconds)
         • SENSE_COLLECTOR_WS_BACKOFF_FACTOR (Default: 1)
         • SENSE_COLLECTOR_WS_RECONNECT_INTERVAL (Default: 840 seconds)

4. Improved Logging:

   • Detailed Logging: Enhanced logging throughout the application to provide more granular insights into the internal processes, including API request timings, cache hits/misses, and error handling.
   • Environment Variable Renamed: Changed SENSE_COLLECTOR_SENSE_API_RECEIVE_DATA_OUTPUT to SENSE_COLLECTOR_OUTPUT_RECEIVED_DATA for clarity.

5. API Worker Enhancements:

   • Improved API Worker Functionality: The API worker now includes more detailed logging, better error handling, and clearer tracking of the processing status for each device.

Functional Changes

1. Configuration Updates:

   • Updated Docker Configuration: Streamlined the Dockerfile to remove unnecessary packages and dependencies, resulting in a smaller and more efficient container image.
   • Removed Packages: python3-dev and gcc are no longer installed, as they are not needed for the current dependencies.

2. Environment Variables Management:

   • New Environment Variables: Introduced new environment variables for managing device data cache expiry, maximum concurrent lookups, and lookup delay.
   • Updated Environment Variables:
     • SENSE_COLLECTOR_OUTPUT_RECEIVED_DATA replaces the old SENSE_COLLECTOR_SENSE_API_RECEIVE_DATA_OUTPUT.

Breaking Changes

1. Environment Variable Changes:

   • Renamed Environment Variable: If you were using SENSE_COLLECTOR_SENSE_API_RECEIVE_DATA_OUTPUT, you must update your configuration to use SENSE_COLLECTOR_OUTPUT_RECEIVED_DATA.

2. Configuration and Deployment:

   • Docker Image: Ensure your deployment scripts are updated to use the new Docker image structure, as the installation process has changed.

Bug Fixes

1. Rate Limiting Handling: We improved the handling of rate-limiting responses from the Sense API to ensure more robust and resilient operation under heavy load.
2. Session Management: Issues with aiohttp session management were fixed to prevent potential resource leaks and ensure reliable connections.

v2.0.03 - New Release

We are excited to announce the release of Sense Collector v2.0.03! This version comes with significant improvements and new features designed to enhance your data collection and visualization experience with the Sense energy monitoring system. Below are the key highlights of this release:

Key Features

1. InfluxDB v2 Support

• New Compatibility: Sense Collector now supports InfluxDB v2, enabling you to take advantage of the latest features and performance enhancements in InfluxDB.
• Streamlined Data Handling: Improved data storage and querying capabilities with the updated InfluxDB client.

2. Rebuilt in Python

• Complete Overhaul: The Sense Collector has been completely rebuilt from the ground up using Python, ensuring a more robust and flexible architecture.
• Enhanced Maintainability: The new codebase is easier to maintain and extend, providing a solid foundation for future updates and features.

3. Low Resource Utilization

• Optimized Performance: Designed to run with incredibly low CPU and memory utilization, making it efficient even on resource-constrained systems.
• Streamlined Sense API Integration: Efficient handling of the Sense API calls to minimize the load and ensure smooth data collection.

4. Improved Logging and Debugging

• Configurable Logging Levels: Detailed logging options to help you monitor and troubleshoot the application more effectively. Logging levels can be configured for general, API, and storage operations.
• Enhanced Error Handling: Improved error handling and reporting to help identify and resolve issues quickly.

Environmental Variables

This release includes several environmental variables to customize the behavior of the Sense Collector:

• SENSE_COLLECTOR_API_PASSWORD: Required, your Sense API password.
• SENSE_COLLECTOR_API_USERNAME: Required, your Sense API username (email address).
• SENSE_COLLECTOR_HOST_HOSTNAME: Optional, hostname for the Docker container.
• SENSE_COLLECTOR_INFLUXDB_BUCKET: Required, the InfluxDB bucket name.
• SENSE_COLLECTOR_INFLUXDB_ORG: Required, the InfluxDB organization name.
• SENSE_COLLECTOR_INFLUXDB_TOKEN: Required, the InfluxDB authentication token.
• SENSE_COLLECTOR_INFLUXDB_URL: Required, the URL of the InfluxDB instance.
• SENSE_COLLECTOR_LOG_LEVEL_API: Optional, sets the logging level for API interactions. Default is INFO.
• SENSE_COLLECTOR_LOG_LEVEL_GENERAL: Optional, sets the general logging level. Default is INFO.
• SENSE_COLLECTOR_LOG_LEVEL_STORAGE: Optional, sets the logging level for storage operations. Default is INFO.
• SENSE_COLLECTOR_SENSE_API_RECEIVE_DATA_OUTPUT: Optional, enables or disables output of received Sense API data to a file. Default is false.

How to Update

To update to Sense Collector v2.0.03, pull the latest Docker image and update your environment variables as needed:

docker pull lux4rd0/sense-collector:latest
docker run --rm \
  --name=sense-collector-53997 \
  -e SENSE_COLLECTOR_API_PASSWORD=CHANGEME \
  -e [email protected] \
  -e SENSE_COLLECTOR_HOST_HOSTNAME=sense-collector.lux4rd0.com \
  -e SENSE_COLLECTOR_INFLUXDB_BUCKET=sense \
  -e SENSE_COLLECTOR_INFLUXDB_ORG=Tylephony \
  -e SENSE_COLLECTOR_INFLUXDB_TOKEN=TOKEN \
  -e SENSE_COLLECTOR_INFLUXDB_URL=http://sense-collector.lux4rd0.com:8086 \
  -e SENSE_COLLECTOR_LOG_LEVEL_API=CRITICAL \
  -e SENSE_COLLECTOR_LOG_LEVEL_GENERAL=INFO \
  -e SENSE_COLLECTOR_LOG_LEVEL_STORAGE=CRITICAL \
  -e SENSE_COLLECTOR_SENSE_API_RECEIVE_DATA_OUTPUT="False" \
  -e TZ=America/Chicago \
  -v /mnt/docker/sense-collector/export:/app/export \
  --restart always \
  lux4rd0/sense-collector:latest

We hope you enjoy the new features and improvements in Sense Collector v2.0.03. As always, we welcome your feedback and suggestions to help us continue to improve the Sense Collector. Thank you for your support!

v1.0.1 - Initial Release

Sense Collector provides a way of collecting real-time data from the Sense Energy Monitor. The Collector also has a set of Grafana dashboards that offer visualizations for detected devices and smart plugs and their wattage, voltage, and amp utilization. Please take a look through the main page readme for details and information on deploying the collector and dashboards.