Raspberry Pi Home Datalogger
Moderators: adafruit_support_bill, adafruit

Re: Raspberry Pi Home Datalogger

by Static on Thu Jan 03, 2013 8:29 pm

A fair warning to folks looking to get their Pi onto a wireless network: you're probably going to have to use an externally powered hub.

I've been trying to get the older Pi set up for wireless. I've got a couple of different wireless adapters lying around (I build a WiFi Sniper Rifle last year), and tried all of them.
The best results I achieved were very erratic link-ups with two adapters, but with nothing else in the USB hub. Plugging in any of the keyboards immediately stopped the device from even being able to scan for networks.

I'm working on getting a USB hub. I'd like to be able to power the Pi directly from the hub (one power cable into the hub, a USB cable from the hub to the PI power port, then a USB cable from the Pi to the USB "in" port), but I haven't found a USB hub that actually powers the ports above 500ma. There are some USB 3 hubs that seem to offer the increased current that I'm looking for, but also at a premium price.
I'm vaguely entertaining a power-hub build where I try to make my own power system and hub. That's difficult right now because my brain hurts from work (seriously people... If you drink, you wind up in the ER. If you stop drinking suddenly you wind up in the ER. Moderation in everything!)
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by tastewar on Thu Jan 03, 2013 10:52 pm

Hmm. I've got a Pi setup with a wireless keyboard and the rarely-in-stock tiny WiFi adapter sold here (product id 814). Powered by just a USB wall wart that is rated for 1.0 amps... Haven't had a problem with it.
tastewar
 
Posts: 352
Joined: Thu Mar 17, 2011 10:16 am

Re: Raspberry Pi Home Datalogger

by adafruit_support_rick on Fri Jan 04, 2013 10:22 am

Static wrote:but I haven't found a USB hub that actually powers the ports above 500ma.

500mA is the current limit in the USB2.0 standard. Many hubs limit ports to that. This hub will supply up to 2A total with no per-port limit: http://www.adafruit.com/products/961


Maybe the problem is in your wi-fi base station, and not in the pi adapters. Have you tried a different base?

adafruit_support_rick
 
Posts: 12793
Joined: Tue Mar 15, 2011 11:42 am
Location: Buffalo, NY

Re: Raspberry Pi Home Datalogger

by Static on Fri Jan 04, 2013 3:27 pm

Am I reading the spec wrong, doesn't that power supply give 5v at 2A. Isn't it much more likely that the hub in turn supplies at most 5v 500ma per port?

What if I used a USB3 hub, with the medusa arrangement of charging and data lines? Is there a fundamental reason that this wouldn't work?

Tastewar, I just put my name up to be reminded when they're back in stock. If a $12 part fixes the problem, that's an easy fix.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by tastewar on Fri Jan 04, 2013 4:17 pm

What are you using the power the RPi? If it's being powered off a computer's USB port or a hub, it probably won't. But my personal experience with a 1A rated wall wart and the above mentioned WiFi adapter, and a wireless kbd/mouse combo, has been positive.
tastewar
 
Posts: 352
Joined: Thu Mar 17, 2011 10:16 am

Re: Raspberry Pi Home Datalogger

by Static on Fri Jan 04, 2013 4:39 pm

I'm using the wall wart that I bought from Element14. I'm pretty sure it's a 1A system.

That's kind of bizarre. I've had tons of problems with wireless keyboards and mice. Repeated characters, dropped clicks, etc. I went to a wired keyboard just to make life easier. Even with the wired keyboard (an old Dell mass produced), I can't get the Wifi systems to come on and stay on. I typically am able to get the networks scanned, but then when I go to establish the connection, the adapter appears to freeze until I remove it and plug it back in.

I'm back at work tonight. I'll probably come home tomorrow and experiment in earnest to find out what the issue is.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by tastewar on Fri Jan 04, 2013 6:21 pm

Just in the interest of disclosure, my wireless kbd is a Logitech K400
tastewar
 
Posts: 352
Joined: Thu Mar 17, 2011 10:16 am

Re: Raspberry Pi Home Datalogger

by Static on Wed Jan 09, 2013 8:42 am

I'm going to experiment a bit today. First, all of my equipment is located in my office. Everything is located within a 2 meter cube. Today I'm going to take one of the Pi's downstairs and hook it up remotely. Ideally I'd like to get it near the front door so I can start the next phase of this project. I'll try the various WiFi dongles that I have and see if I can get it up and running. I'm also going to hit Home Depot and build a suitable outdoor enclosure for the sensor cluster. I've got an interesting idea for this, plus I'll be able to get the cluster partially shaded.

I'm working on using the WebIDE more. I need to figure out how to get my current files over to the repository. Ideally I just want to set up a directory shortcut or pointer... I think. I read about doing this somewhere in the last couple of months.

Finally, I put VNC on the remote Pi. I was looking for something that I saw on the Adafruit Tutorials. Coincidentally, the Pi Tutorials are NOT located under the "Tutorials" tab of the Adafruit main page. You guys should look at fixing this. You've done some incredible work here, make sure people can find it as easily as possible:
http://learn.adafruit.com/category/raspberry-pi

I've got more I2C equipment coming. I just saw that there are some new tutorials in the Adafruit section on the GPIO. I need to play with that.

I need to figure out how to safely get my remote nodes to come up after a loss of power. I don't want to have to hook up a keyboard (and probably a monitor) each time these things go down. Is there a (relatively) safe way to have them boot-up without a password? There has to be a way to limit their network connectivity. Eventually, my plan is to have all of the nodes talking to other nodes in the system, with just a couple of those nodes (PC's, tablets, phones) able to communicate outside of the network. Those externally communicating nodes would all have their own security procedure (IE: My login and password on my desktop and tablet. My unlock procedure on the phone). Any ideas? I'm not looking to serve up a zombie-net on a platter. I'd like to be thinking about the security aspects from the start. Let me rephrase that: I'd like to be dealing with the security aspects from the start.

Tastewar, I'll post up my equipment later today. I don't want to wake the dogs. My GF is getting to sleep in for once.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by Static on Wed Jan 09, 2013 5:10 pm

I'm using a wireless Logitech MK320 keyboard and mouse set.
When I'm wired I'm using a Dell SK-8115 keyboard (it's the keyboard that is in front of every generic Dell office computer).
I was using an Airlink 101 miniature USB 802.11 nub.
I've switched over to a Ralink....something. It's from Monoprice. I got both the WiFi adapters when I was building a WiFi sniper rifle.

I was playing around this morning and noticed something peculiar during boot up. The system was giving me some kind of error configuring wlan0, so it renamed it to wlan1. I went and played with my "interfaces" file, and added an entry for wlan1. Bam, the Ralink system worked.

Now I'm running into an issue with the wireless keyboard. I think I have too much 2.4ghz radio energy in my office (I haven't noticed a tan... yet). The keyboard misses keystrokes or generates key stuck issues. I'm missing mouse clicks, etc.

I need to grab some more 16GB memory cards. I can't seem to clone my working card to a smaller card. That's going to be a bit. I just paid all my bills (well, most of my bills), and the responsibility has killed my bank account.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by Static on Thu Jan 10, 2013 11:35 am

Alright, change of plans. I had a dream last night that made me realize a couple of things (yeah, I'm geeky).

Cloning the cards over is going to clone all of the "bad things" that I did to the distro. The card contains all of the flailings that I inflicted on the OS, Python distro, and tweaks that occurred while I was learning Linux and re-learning Python. I'm going to take the time to do it right, and I'll clone that setup as well.

A couple of quick notes:
First, if you're setting up a US keyboard, make sure you de-select the UK default keyboard. I didn't realize that the UK keyboard would take precedence in the set-up. I've been going through and altering text files.

Some of the steps on here take some time, like more than 5 minutes (I don't think anything here takes more than 10 minutes over a wired ethernet connection). Go get a cup of something and relax.

Don't forget to update your apt-cache and Raspi-config

Next, an issue that I keep having: Force HDMI output (My monitors are all HDMI capable. Powering off the board causes the video output to swap between HDMI and composite).

Code: Select all | TOGGLE FULL SIZE
cd /boot/
sudo chmod 777 config.txt
nano config.txt

##########De-comment two lines (Remove the "#" character in front of them)
hdmi_force_hotplug=1
#######and
hdmi_drive=2




Now, this will present an issue if you ever want to boot the system with a composite video screen. I'm planning on using SSH and VNC to get into these guys

At this point I rebooted to get the screen real estate that comes with an HDMI connection.

Next, I'm installing the VNC server:
http://learn.adafruit.com/adafruit-raspberry-pi-lesson-7-remote-control-with-vnc/installing-vnc
For some reason I keep typing an extra space when trying to start the VNCserver. It should be like this:
Code: Select all | TOGGLE FULL SIZE
vncserver :1

After following the tutorial I'm still having to start the vncserver manually. Dunno. YMMV

I'm taking the time to copy my PiLog files over from the USB stick to the newly configured Pi. I'm ONLY transferring the code files for Python. I'm still going to need to configure the Apache server files (I think), and pull down some of the libraries/modules for Python3.

Next step: WiFi access. I'll warn you, this has been a little iffy on my end and one of the things that prompted me to start fresh.
Here's the Adafruit tutorial (Dude, Simon Monk, is there anything you can't do?):
http://learn.adafruit.com/adafruits-raspberry-pi-lesson-3-network-setup
Some words of caution: For ease of use, examine your USB WiFi links early, and decide which one you want to use. If you try multiple USB links and one doesn't work, you may have some extra steps.
I preferred to manually configure the "interfaces" file using nano.

Before rebooting, I'm installing the Adafruit Raspberry Pi WebIDE:
http://learn.adafruit.com/webide
While I'm talking about this, let me underline how much SSH is your friend.
I'm using putty, from Sourceforge: http://sourceforge.net/projects/putty.mirror/?source=directory
These guys did good work. I was pointed to putty when I was working on an Arduino project that needed some good serial logging capability (Arduino smart trigger for an airsoft gun).
Using SSH from my main PC (I'm a Windows guy), I can cut and paste commands from the tutorials and right click them into my SSH session. Typing out the command to get the WebIDE onto my Pi would have been fraught with problems.

When I first learned of the WebIDE I tried to jump right into it. Unfortunately I created a BitBucket account and started reading their tutorial. My ADD took over and I went down this tangent for several hours, configuring my computer, tweaking things, moving things around, and breaking stuff. I spent the last half of the day unbreaking things (I think. We'll find out today). I wound up not using the WebIDE. Hopefully this attempt will get it correct.
Interestingly enough, the report back from the installation file says that the server is running on port 3000. It's actually working on port 80. This may create problems for me down the road when I try to install the Apache Server and expose the port to the internet (so I can see the data collection when I'm not home). Right now, I'm just going to go with it.
I did just change the hostname and port using the tools available at the WebIDE page. Checking my SSH terminal by issuing the command "hostname", it appears to have changed correctly. I'm rebooting the whole Pi to see if the port settings changed. It took two attempts to log on, but it's changed.
After that, I continued the tutorial, generating Customers and Keys. At this point I'm happy with where I'm at with the WebIDE, so I'm going to push on.

I hooked up my sensor cluster to the Pi Cobbler board (the first version. I'd recommend the T version of the board if you're prototyping. If you have a semi-finished project, I'd go with the inline board. I'll post some pictures later).
Now I want to check the I2C bus because my project relies very heavily on it. I'm using the first version of the Raspberry Pi Model B board (the lower memory version).
Code: Select all | TOGGLE FULL SIZE
sudo i2cdetect -y 0

The second version of the board (The 512mb board) uses this command:
Code: Select all | TOGGLE FULL SIZE
sudo i2cdetect -y 1

This code generates a grid of addresses. The "Tens" column of the address is to the left. The "Ones" column of the address is across the top. If an I2C address is being used, the system reports the address at the intersection of the "Ones" and "Tens". Example: My board has the following I2C addresses in use: 1e, 48, 53, 69, 77. On the grid, at the intersection of "10:" and "e" the address is reported as : "1e". At the intersections for the other addresses the others are reported. Everywhere else a simple "--" is reported.
If you i2cdetect doesn't generate the results you expect, swap the final number (a 1 for a 0) and see what happens.
I have "crashed the bus" by messing around with the connections on the board, and I think by polling sensors too rapidly. Again, YMMV.

Relocating the Raspberry Pi, sensor cluster, USB WiFi, and power supply to the other side of the house took some finagling. First, I'm really glad that the Monoprice USB dangle is not a dongle (if that makes sense). It's allowing me to put the antennas remote from the Pi. This is going to help, as the wireless environment around the townhouse is busy. My S/N ratio drops precipitously.
I'm able to VNC over with a little lag. I'm trying out some of my code and installing the libraries (are these called dependencies?). Right now I'm working on matplotlib.
I also have to remember to change the Adafruit_I2C.py I2C bus back to 0. It was set to 1 when I switched over to the Version 2 Pi.

OK, I do need to reinstall Apache.
Code: Select all | TOGGLE FULL SIZE
sudo apt-get install apache2

It takes a little bit to install Apache, be patient.
When you get done with this, type in your Pi's IP address from another computer. You should see the Apache test page.

And that's it! My basic code is up and running. It looks like all it needs is matplotlib and Apache.

OK, that's enough for today. I'm going to try to get the board set up remotely on the otherside of the house. More tomorrow. I'm going to work on getting the repositories publicly viewable so that anyone can grab them. I also need to get some serious picture documentation done.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by Static on Fri Jan 11, 2013 6:30 pm

The previous post was probably TL:DR. I need to document somewhere... Maybe it will help someone else.

Code: Select all | TOGGLE FULL SIZE
#!/usr/bin/python
import time
####    import the sensor modules here
from GPIOState import PinCheckSingleBin as MotionSensor

####

print("Start")   
MotionTally = 0
MotionDetect = 0

def scheduler():
    currentTime = time.asctime()
    currentSecond = time.localtime().tm_sec
    currentMinute = time.localtime().tm_min
    currentHour = time.localtime().tm_hour

    if currentSecond == 01:
        MinuteOne()
    if currentSecond % 6 == 0:
        SixSeconds()
    if currentMinute == 01:
        HourOne()

def MinuteOne():
    ###Tasks to perform every minute.  Put the tasks below
    print(MotionTally)
    MotionTally = 0
   
    time.sleep(1)  ###This delay may need to be altered
   

def SixSeconds():
    ###Tasks to perform every six seconds.  This is used to poll the motion sensor
    MotionDetect = MotionSensor(16)
    MotionTally = MotionDetect + MotionTally
    MotionDetect = 0

    time.sleep(1)   ###This delay may need to be altered

def HourOne():
    ###Tasks to perform every hour


    time.sleep(0.5)   ###This delay is set so that additional tasks during the
                        ### The top of the hour can be done


while 1 != 0:
   
    scheduler()


Code: Select all | TOGGLE FULL SIZE
1
Start
1
Traceback (most recent call last):
  File "Scheduler.py", line 51, in <module>
    scheduler()
  File "Scheduler.py", line 21, in scheduler
    SixSeconds()
  File "Scheduler.py", line 36, in SixSeconds
    MotionTally = MotionDetect + MotionTally
UnboundLocalError: local variable 'MotionTally' referenced before assignment


I'm getting an unexpected output, in a couple of different ways.
The initial "1" that is returned before "start" is throwing me. It should print "Start" right as the program is being initialized.
Immediately after printing "Start" it should initialize MotionTally. Instead, it's not, and it's throwing an error.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by Static on Sun Jan 13, 2013 4:56 pm

I'm working with a friend of mine who is prodding me a bit. He's given me some excellent suggestions and shown me some places I can do better.

The project is up and running. It's logging data from it's sensors and putting up a webpage with historic charts.

I'm going to work on cleaning some of it up. First, my friend pointed out some problems with my scheduling code. He's got some tips for making it cleaner and more consistent. That's largely done, but not implemented. What I'm working on now is creating a generalized Sensor Class in Python, something that will handle getting sensor information from the Pi and presenting it to code. I'm really having trouble wrapping my head around everything. There are a couple of things that I want to make easier.

1. Make sensor designation easy. When I have an indoor thermometer I want to give it a name and assign all the software "stuff" to behind the scenes. When I call the name in the software it presents me with a number or a true/false.
2. I want to be able to use the same sensor different ways. Example: Ordinarily I am passively interested in the temperature in the house. If I want to see it, I'll go look for it. I'd like to see the specific number and a graphic trend. However, if the temperature gets above 30 degrees C, I want to know about it immediately. I don't care what the actual number is (except for debugging purposes, but I've already got a log for that). Here, I'm using the same sensor, but in two different ways. I'd like the code to be able to handle this transparently.
3. I want to be able to have the code check for issues. I2C address conflicts, digital pins that are apparently assigned to the wrong pins, I2C addresses assigned to digital pins, etc.
4. Eventually I want the code to look beyond a single Raspberry Pi board (IP network based sensors). Right now the system is two Raspberry Pi's working over an ethernet and WiFi network. I'd like the Pi's to be able to examine the sensor input from other Pi's (almost any number in a local network). This would allow for some pretty neat feats. I wake up and take my shower. The bedroom Pi sees that I'm awake and turns the thermostat up. I come out of the shower and shake my phone awake. The banned PI sees this and turns on my coffee maker. The house network sees this and turns the internal motion sensor portion of the security system off.

It's a straight forward project, and I think I've defined it well. Execution is going to be the big deal. Fortunately, I realize I can start with the basics and start hanging more code onto the basic portion of the code as I move forward.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by Static on Sun Jan 20, 2013 1:47 am

I worked a little bit on the I2C TSL2561 luminosity sensor:
http://forums.adafruit.com/viewtopic.php?f=8&t=34922&start=15
Admittedly driverblock and pandring did all the heavy lifting. I'm not even sure my contributions are correct. The sensor seems to act peculiarly when I put large amounts of light on the sensor.

I've been poking at the Sensor Class work that I've been playing with. Right now, it's a mere skeleton. Looking at it compared to all of the sensors that I'd like to be able to read, it's a skeleton that's going to need to do an incredible amount of contortion before it's really usable. I'm trying to wrap my head around whether it is better to create a Sensor Class like this, or to compile a single library with as many I2C sensors for the Raspberry Pi as possible. It would help if I had more than a day at a time to work on this.

I need to order more parts (ugh!). Now that I've got a working version of the TSL2561 sensor, I want to get an additional one to put in the outside sensor pod.

In that vein, I upgraded the sensor pod. It's no longer a lemon juice bottle on the end of a CAT5 cable. It's now a properly vented enclosure, with a clear dome (a small tupperware) on top for the eventual light sensor (the TSL2561). I'll post some pics and a build guide later this week. It's going to be a busy for us emergency medicine guys in DC.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by Static on Sat Feb 09, 2013 8:40 am

I've started a massive re-write of my code. Specifically I'm in the process of rewriting my data-logging code to make better use of the Python Dictionary data type and the cPickle module. It's an incredibly primitive portion of the project, but it has surprised me how much skull-sweat it has taken. I've gone back and forth about the best way to implement the log, and I think I've hit on it. Ultimately I went with what is probably the easiest way (which makes me feel pretty foolish now).

Initially I was going to go with a giant dictionary of a dictionary, or a list of dictionaries. Each entry in the log would be a dictionary. Each datavalue in the log would be keyed to a data identifier. The log would also store the epoch time (the number of seconds that have elapsed since 1/1/1970), a time date stamp, and a logging level value similar to the values found in the Python Logging Module. I stopped using the Logging Module because it didn't appear that I could store an actual dictionary with the log. I was slicing the logs like crazy, and it made for some significantly ugly and hard to understand code. I was doing some incredible workarounds that weren't simple. You might have been able to call them clever, but that's not really a complement in the Python parlance.

The dictionary of dictionary approach and the list of dictionary approach were overly complicated. I actually did code the dictionary of dictionaries. It was ugly and complex. I figured out that using cPickle would allow me to automatically index each dictionary/log entry.

Right now I'm coding functions to pull data from the log in a coherent fashion. Ultimately I want to be able graphs, webpages, and other reports from the new logger module. My last logging efforts had some issues in complexity and error propagation. I need to keep an eye on that, and create functions for correction and validation. I think this new system will also allow me to code for long term files. I'll be able to archive very old data while preserving the more current data, automatically. This will let me keep the limited space on the Pi for relevant data.

It's bizarre as hell to think of 8-16GB as limited space.

I've been working on two different sensor configurations. I think some of the code inconsistencies came from working on two different configurations. This violates one of my initial goals, of building a system that is tolerant of different configurations (open source, easily modified). This new code should be much more tolerant of different conditions. The code will look for values associated with a key, and if the key isn't there the function will either create a specified garbage/filler value, or will skip log entry entirely. This was a major problem with the previous code version.

Gotta keep plugging away at it.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm

Re: Raspberry Pi Home Datalogger

by Static on Sun Feb 10, 2013 2:51 am

(Insert random scream and sailor swearing)

I spent the day generating almost 5,000 records from the sensor system.
I just sat down to see how the new code handles generating a chart. It's taking minutes, more than 3, to generate a chart. Granted, I'm pulling 4,000 datapoints from almost 5,000 lines of log, but that's still insanely slow. The performance hit is at least 3-4 orders of magnitude worse than the previous system.

I thought cPickle was going to be much faster. I need to look and see where I screwed up.
Static
 
Posts: 180
Joined: Thu Dec 23, 2010 6:21 pm