ZIG2410 sleepy mesh question

[RoSz]

We are using CEL's ZIG2410 and started up with the LowPowerMeshRouting pdf.
Unfortunately we got stuck with the first scenario. It seems that the sleep - wake cycle doesn't work as intended.
We tested the first scenario just right out of the box with default values (5+2+1), the only modification is that we removed the setSegments code from sleepymesh.py (since CEL eval board doesn't contain a 7 segment display).
The thing we noticed was that after loading the sleepymesh.py onto the board (following the instructions in scenario 1), the module seems to be awake for about 9 seconds (almost 9 blinks), then led is turned off for about 1 sec and then led is light up for about 5 seconds. This is not what we expected.

Question: Does this SNAP samples (referenced by LowPoerMeshRouting pdf) not apply to the CEL modules (and the ZICM2410-EVB3 evaluation boards)?
Or have we misunderstood in some way?

(We are using the 2.2.14 SNAP core)

[kbanks]

Not having a seven-segment display is making it trickier for you to follow-along with what the node is doing.

The 5-2-1 refers to the timing, once the node starts "cycling".

When you first power it on, it waits a bit to see if there is already a "leader" node. It then holds an election (by itself). Being the only node, it "wins" the election, and then begins leading a "mesh of 1".

The seven-segment display showed indicators of "L?", "=L", etc. so you could tell what state the node is in.

That being said, all of the application notes predate the 2.3 SNAP ports (including Panasonic and CEL), and I don't know if their example scripts need any modifications or not.

Updating the application notes for 2.3 is towards the end of the list. Currently we are working on the EK2100 and EK2500 User Manuals, as well as a new Portal manual.

[RoSz]

Ok, thanks.

Instead of the 7-segment display, we use 4 leds (default on ZICM2410-EVB3).
First one blinking when awake, second when election takes place, third when the module won the election (=leader) and forth if module is a follower.
The situation is (with one module activated only) that an election takes place (led 2 for about 2 sec) then election is won (led 3 lights up very shortly), then all leds shuts down (as it goes into sleep mode) BUT after about 1 sec all leds will go high. In my opinion it seems there is a reboot taking place or something at this point, prohibiting the module from beeing shut down 5 seconds.
Of course I could be totally wrong.

As I understand the code should work for 1 node (mesh of 1) as well as for several?

If we activate another module, it seems that the communication takes place (2:nd node will become follower) but the same strange "reboot?" takes place at both modules (all leds goes high).

Please advise if this question is for CEL or if Synapse could help us out?

Thanks!

[kbanks]

after about 1 sec all leds will go high.

There's the clue we needed!

What you are running into is that the sleep modes are different between the CEL module and the 9S08 based ones.

On the 9S08 based modules, mode 1 is the best, because it is the most accurate (and there is only a mode 0 and a mode 1).

Keep in mind that all the app notes were written back when this was the ONLY kind of SNAP Engine available.

On the CEL ZIC2410, there are three sleep modes (0-2), and only mode 0 retains I/O settings through sleep. Mode 1 "de-powers" the IO drivers (this is why your LEDs are not showing the correct values after about a second of "discharge time".) Mode 2 does not drive I/O either, plus it RESETS the entire module at the end of the sleep cycle (making it only useful for specialized applications).

You can read more about the ZIC2410 sleep modes of the ZIC2410 in it's datasheets. The important thing to know if the "offset" between the modes as presented in the datasheet, versus the numbering we use in the sleep() built-in.


CEL Datasheet SNAPpy sleep()
=========== ===========
0 CEL Power Mode 0 is "awake", and so does not apply
1 sleep(0)
2 sleep(1)
3 sleep(2)


Try using sleep(0, xxx) instead of sleep(1, xxx) in your test scripts...

[RoSz]

Super! That solved our problem, Thanks!

[RoSz]

Another question that arised was the synchronization between the leader and the followers in the sleepy mesh.
As we set up the original example of sleepy mesh, it seems that the leader wakes up at the same time the followers are going into sleep (we observe this with the led's).
As we have understood, all nodes (both leader and the followers) should be synchronized and have the same sleep-wake rythm.

Or perhaps a misunderstanding from our side?

[kbanks]

all nodes (both leader and the followers) should be synchronized and have the same sleep-wake rythm.

Yes. You may have allowed some other node to become the leader (to become an additional leader). Try multicasting a reboot() command to sync everybody up.

[kbanks]

Be sure to notice that there are TWO Sleepy Mesh App notes (one is newer).

Be sure to look at the newer one, too. I would hate for you to be struggling with something that is already covered in the newer app note.

[RoSz]

Thanks for the update.

We have downloaded the second app note and uploaded the sleepyExample on each node and sleepyMonitor on Portal. There are several things that seems strange but I start up with the major ones.
1. As I mentioned in previous posts after the net is in sync (all followers are in sync) it still seems that the leader wakes up exactly when the followers go to sleep. We use led's to display leaders (green) and followers (blue) but the green led is lighten up exactly (or perhaps some milliseconds before) when the blue is shut down. As we have understood, all nodes (both leader and followers) should have totally the same sleep-wake rythm? (From this assumption both the blue leds and the green led should have the same rythm). This is what we could see a major problem since the message transfer (we will need several hops in our net) will not work (at least not from the leader node and since the leader will be dynamically chosen, any node could be prohibited from sending messages).
2. With the new appnote examples (sleepyExample), sometimes two leaders are elected and they will get totally out of sync and the net will remain having two leaders. This is not anything we have seen in the previous examples and I think we will need to examine the difference between the scripts a little bit more before we could say anything about it.

[kbanks]

1. As I mentioned in previous posts after the net is in sync (all followers are in sync) it still seems that the leader wakes up exactly when the followers go to sleep.

In this particular case, how many leader nodes do you have? (Should only be 1 leader)

The followers should not be going to sleep until told to by the leader, so I don't understand "leader wakes up when followers go to sleep".

[RoSz]

In this particular case, how many leader nodes do you have? (Should only be 1 leader)

We only have 1 leader.

The followers should not be going to sleep until told to by the leader, so I don't understand "leader wakes up when followers go to sleep".

Ok, so perhaps misunderstanding from me. Does it make sense that when leader wakes up, he immediately sends signal to followers to go to sleep (since there is nothing to process)?

[kbanks]

Does it make sense that when leader wakes up, he immediately sends signal to followers to go to sleep (since there is nothing to process)?

Depends on the script loaded into it. Most of the example scripts support the concept of an "awake time", which postpones the next "sleep period" and gives you time to actually do something with the units.

As a sanity check, is there a "spare" LED you can blink from the HOOK_100MS handler? (use pulsePin). I sometimes use such a "blinker" to help confirm when nodes are actually awake versus asleep.

A static "status" LED might get set to the wrong value due to a scripting mistake (turned off when you meant to turn it on, or vice versa). Or you might forget to change it before going to sleep, or after waking up.

A "blinker" on the other hand will only blink when the node is awake. The timer hooks do not run when the node is asleep, so "solid on" or "solid off" on the (normally blinking) LED is a reliable indication of sleeping.

[RoSz]

We are using blinking led's (each 100 ms) so it's clear when the nodes are awake or not.

The only question I would like to have an answer on at this point is if the leader is supposed to be up 2 seconds after the followers or not. If not, I think we might have some major problems. (we are using sleepyExample.py and have only modified the code so it uses led's instead of 7segment display).

[kbanks]

if the leader is supposed to be up 2 seconds after the followers or not

Everybody is supposed to wake up together.

The Leader is supposed to "count up" to the duration specified by variable wakeTime. Look at variables wakeTime and wakeTimer in the scripts.

The Leader is then supposed to broadcast the "everybody go to sleep" command.

Everybody is supposed to go to sleep together.

(this should repeat)

Everybody is supposed to wake up together.

The Leader is supposed to "count up" to the duration...

[RoSz]

...this is not happening :(

It seems that the leader sends out a multicast rpc so every follower goes to sleep and that's fine. But the leader should go to sleep directly after but this is not happening.

So this is what's happening when the leader has counted up the wakeTimer to 2000 ms:
1. In sendSleep function it does a multicast rpc to invoke method gotoSleep (all nodes will receive this and it seems that this is actually happening since the followers will go to sleep)
2. It then calls doSleep that first invokes beforeSleep (this function only sets the 4 leds low) and then it invokes sleep(0, dur) where dur is sleepTicks (this is set to 5 as default) and the '0' is set since we use ZICM2410 and we need to use '0' as sleep mode in order to persist GPIO state after a sleep cycle.
Note that we also have set the calculated sleepTime to be sleepTicks * 1000.

So what we could see the problem has to do with the fact that the leader won't go to sleep directly after it has sent the go-to-sleep command to all other nodes. Since it is awake for 2 seconds after it has sent the go-to-sleep command to the others, it will be somehow out of sync.
But again, we could be wrong on this conclusions...

Another problem we see regulary when rebooting the nodes is that there is elections taking place now and then. And unfortunately sometimes the nodes get out of synch so there could be 2 leaders.

And we have also seen several times that the nodes could hang. This has happened a couple of times during sleep (the module did not wake up again, at least no leds where lighten) and twice during an election.
This is very serious.
Have you seen this kind of problems on the CEL modules and is there a chance that Synapse could test these scripts on the ZICM2410 evaluation boards as soon as possible so these kind of problems could be sorted out?

[kbanks]

invokes sleep(0, dur) where dur is sleepTicks

Please use a print statment or an rprint() RPC call to verify that variable dur is non-zero at the line in question.

SNAPpy ignores sleep(0,0) unless a wakeup pin has also been programmed.

[RoSz]

Hi,
We have printed out the dur as suggested and it's 5 (as it should be).
Nevertheless we found something that might give a clue on what's happening:

def doSleep(dur):
"""this is the actual "going to sleep" code"""
global isLeader, wakeTimer, electTimer, sleepMode

beforeSleep(dur)
print 'beforesleep'
sleep(0, dur) # 0 is ZICM2410 sleep mode used (@ 25 uA)
print 'aftersleep'
afterSleep()
...

Above is the code is the default example code except for that we are setting sleep(0, dur) - '0' since we use ZICM2410.
We also added print statements beforesleep and aftersleep.
Strange thing is that when we intercept the stdout in Portal, both beforesleep and aftersleep comes in one occation:

2009-10-08 10:52:18 RS: beforesleep
aftersleep
2009-10-08 10:52:18 nodes are asleep
2009-10-08 10:52:18 nodes are awake
2009-10-08 10:52:45 RS: beforesleep
aftersleep
2009-10-08 10:52:45 nodes are asleep
2009-10-08 10:52:45 nodes are awake

(RS is the name of our gateway node)
So above is what is happening with only one node (gateway) running. We were expecting that beforesleep should be printed out and then the module should go to sleep, then when it wakes up the first thing would be the aftersleep print out. But on the contrary, both beforesleep and aftersleep comes at the same time.
Question: Is there some latency involved with the sleep function so the module won't be brought into sleep mode directly?

(Thanks for all help we are getting - we thought it would be an easy piece getting the basic app running but apparently we were quite wrong)

[kbanks]

Question: Is there some latency involved with the sleep function so the module won't be brought into sleep mode directly?

No. There *IS* the "disallow sleep with no hope of waking up" check I already mentioned, but the sleep call (if allowed) is immediate.

That is a good question, though... Built-in reboot() would be an example of what you describe. The built-in only schedules the reboot, which happens later.

Since you've confirmed the duration is non-zero, my next guess is that "pin wakeup" is enabled, and that instead of "not going to sleep", what is really happening is a short "go to sleep, get woke up by EXT0 and/or EXT1".

You could try importing the appropriate pinWakeupxxx file (look in the synapse subdirectory) and explicitly disabling the two "wakeup" pins.

[RoSz]

Hi,

Tested to apply the disabling of wakeup pins as you proposed according to the following:

def atStartup():
"""This helper routine is invoked from the real startup handler"""
global sleepTicks,wakeTime
sleepTicks = 5 # this one represents "ticks", as used by sleep()
# The rest of these are in milliseconds
wakeTime = 2000
initUart(0,0)
setPinDir(ledPin1,True)
setPinDir(ledPin2,True)
setPinDir(ledPin3,True)
setPinDir(ledPin4,True)
writePin(ledPin1, False)
writePin(ledPin2, False)
writePin(ledPin3, False)
writePin(ledPin4, False)

wakeupOn(GPIO_18, False, False)
wakeupOn(GPIO_19, False, False)


but it's still the same problem (beforesleep and aftersleep print statements comes out at the same time).
The idea with the wakeup pin is also not true in my opinion, since we experience that the module actually goes to sleep for 5 seconds BUT not where we expect it (in the sleep function call). Instead for going to sleep when we call sleep function, it goes to sleep 2 seconds AFTER this call has been made :confused:

I am really out of ideas on what the problem could be. If you could confirm that the sleep(0, 5) should work as expected (put ZICM2410 into sleep mode during 5000 milliseconds and then wake up with GPIO's restored), then in my opinion this must be a defect in the SNAP core.
Could you please also confirm if there are other customers using SNAP with ZICM2410 with the sleepy mesh functionality?

Thanks!

[RoSz]

Hi,

I added a print statement of getMs() in the beginning of the timer100msEvent.
The following could be observed:

2009-10-08 18:34:30 RS: 11023
2009-10-08 18:34:35 RS: 11114
beforesleep 11114
aftersleep 11114
2009-10-08 18:34:35 nodes are asleep
2009-10-08 18:34:35 nodes are awake
2009-10-08 18:34:35 RS: 11206
2009-10-08 18:34:35 RS: 11297
2009-10-08 18:34:35 RS: 11388
2009-10-08 18:34:35 RS: 11479
2009-10-08 18:34:35 RS: 11570
2009-10-08 18:34:35 RS: 11661
2009-10-08 18:34:35 RS: 11752
2009-10-08 18:34:35 RS: 11843
2009-10-08 18:34:36 RS: 11934
2009-10-08 18:34:36 RS: 12025
2009-10-08 18:34:36 RS: 12116
2009-10-08 18:34:36 RS: 12207
2009-10-08 18:34:36 RS: 12298
2009-10-08 18:34:36 RS: 12389
2009-10-08 18:34:36 RS: 12480
2009-10-08 18:34:36 RS: 12571
2009-10-08 18:34:36 RS: 12662
2009-10-08 18:34:36 RS: 12753
2009-10-08 18:34:37 RS: 12844
2009-10-08 18:34:37 RS: 12935


The first line is the last line written before the module actually falls asleep (for 2 seconds).
The second line is the first line when the module wakes up.
The rest of the lines are occuring each 100 ms (according to 100 ms event since the print out is in the beginning of this function).
The last line is just before the module actually falls asleep again.

The interesting thing is that it is 91 ms between each printout??? Isn't that strange? Shouldn't it be 100 ms between the printouts?