1

We have a number of IoT devices that communicate over TCP/IP to a server. Once connected, the server can then talk to the device, get data or send down commands.

The server accepts incoming connections from hundreds of IoT devices on a specific port.

My question is around testing of new code and deployment. Heres my scenario. There is a bug with one of the devices acting strangely or doing something wierd. Now normally you would try and replicated the bug in a test environment. But the thing is this is very hard to do. So what I end up doing is debugging code on the PRODUCTION server and step through the code for that particular device.

You see most of the bugs are hard to replicate in a test environment. You actually need to debug code on the production server connecting to a live device.

As you can imagine this is not ideal but if a particular device is playing up in the field, you want to be able step through the code with that device.

At first I thought I could divert the traffic of this particular IP to a test server with Visual Studio installed and then take my time to debug the problem. Only you cant divert a single IP. You can only port forward a whole port to another server. This means the whole estate of IoT devices get sent to the debug server.

I would like to hear how other people would go about this problem? Is there another way to single out an IP and debug it in isolation without affecting the other production devices. Or is there another way of approaching this problem all together. (By the way, these devices are geographically spread across the country which means we cant just go and change the IP on site)

Obviously this method plays havoc with change management as I am constantly putting debug code on the server!! - Instead of having the luxury of solving the problem on a test server I am messing with production!

Is there a better way of approaching this problem? Any thoughts would be very much welcome and thanks in advance.

VirtualBrandy
  • 11
  • 2
  • How do the IoT devices know what IP address the server has? Is there a way to get a sample of a device that causes problems into an environment where you have more control over the network? – Bart van Ingen Schenau Dec 03 '17 at 18:42
  • The server has a static ip address that the IoT devices use. No there isn't. The only way I can think about doing thst is by changing the server ip address the device connects to. – VirtualBrandy Dec 03 '17 at 19:00
  • You need a separate staging environment for testing where you deploy a test version of the server as well, with the same IP address as the production server (but on an isolated network). – James McLeod Dec 03 '17 at 19:09
  • So are you saying the devices in the field would somehow connect to this staging server when there was a problem. The big question is once a device is deployed to the field and a problem is found how would you push this device over to the staging environment? – VirtualBrandy Dec 03 '17 at 19:43
  • Exactly why couldn't a proxy assume the IP address and divert some traffic to production and some to testing according to how you program it? – candied_orange Dec 03 '17 at 21:11
  • Hi Candied Orange. I like that idea (can't believe not thought of that) but can a proxy server divert a specfic ip addresses? We are on windows. What proxy server do you recommend? – VirtualBrandy Dec 03 '17 at 21:35
  • You said that server can send commands down to the devices. Any chance to set up a new IP address to an specific device by commands? Do you upgrade devices software? I mean, any chance to deploy an upgrade for you to allow the server to configure devices remotely? – Laiv Dec 03 '17 at 21:46
  • Hi laiv. Unfortunately not. These devices are not upgradable remotely. They are upgraded but someone physically has to upgrade it. – VirtualBrandy Dec 03 '17 at 22:13

1 Answers1

1

Is there another way to single out an IP and debug it in isolation without affecting the other production devices.

Use a firewall or modify the routing table of the production server to route the traffic coming from the IP address. For example:

sudo route add 4.2.2.4 -interface <eth2> # reroute traffic from 4.2.2.4

cmd 00300 deny all from 4.2.2.4/16 to any in via $pif # deny traffic from 4.2.2.4

References

Community
  • 1
Paul Sweatte
  • 382
  • 2
  • 15