NSLog.exe

Version 1.41 July 19, 2011

NSLog is a small utility program that reads the NexStorm™ Flashgate IPC-1 and 2 memory areas and then creates files suitable for enabling javascript updates of web pages as well as the creation of maps and graphs. It also has the option to upload several files at user-defined rates, which will then be accelerated when a user-defined level of lightning activity is reached. Files are only uploaded if they are newer than the previous copy uploaded, thus saving bandwidth to be better used when there is high activity. A great example of what can be done with this data is the NSDisplay script that is available on the Add-Ons tab.

NSLog was written using C++ Builder 6 and is based on the code provided by Astrogenic Systems (Relko) and included on the NexStorm™ distribution disk.

If you find this program useful and you wish to help defray the cost of the compiler, please feel free to use this link.  

Install/Uninstall - Download and unzip NSLog.zip, then place the NSLog folder anywhere on the hard drive of the computer running NexStorm™. To uninstall, simply delete the NSLog folder.

NSLog.zip contents - The NSLog folder contains:

1) NSLog.exe
2) nsFtp.exe (optional use now)
3) readme.txt

Operation - Simply run NSLog.exe and it will begin reading NexStorm™ data from the Flashgate IPC areas. Upon running the program for the first time, there will be a NSLog.cfg file (and backup) created to maintain your configuration data. There will also be a "hidden" \nslupload\ folder created which will contain the most recent copies of the files that are uploaded. Please don't set NSLog to create any of it's files in this folder! If starting NSLog with a shortcut or similar, be sure to specify the \nslog folder as the "start in location" or "working directory" so that its files are created in the right place.

There is also a command line switch "m" available to start NSLog minimized to the system tray (nslog m).

Menus

File - Clear Screen - Clears the scrolling screen display.

File - Minimize to Tray - Minimizes the program to the system tray. Clicking on the "-" in the upper right corner of the program will do the same.

File - Exit - Exits the program and saves the current configuration. The red x in the upper right corner of the program screen will do the same.

View - View xxxx File - Opens Notepad and displays the various files..

Config - See Config Tab.

Help - Online Help - Will open your default browser and bring you to this page.

Help - About - Displays the About screen which has version and contact information.

Config - General Tab - This screen has several options for configuring the program.

Screen On - Controls whether the data is written to the screen. If you're having trouble and have a very slow computer, you may want to experiment with turning this off.

Show in Miles/Km - Allows you to select the units that are displayed as well as logged.

Show Heartbeats & Noises - Controls whether these are written to the display screen.

Show Status Area - Show or hide the lower strike and upload information.

Confirm Red X Shutdowns - Whether you want to confirm program exit when clicking the "Red X".

Log Raw Data - Controls whether the raw data is logged.

Include Heartbeats & Noises - Not normally used, but available should they be needed.

Archive Raw Daily - With this checked a new file is created each day at midnight. Files are named yyyymmddNSRaw.txt.

Append Recap - If checked, a short recap of the day's activity will be appended to the end of NSRaw.txt when it is archived.

Archive Folder - Allows you to designate where all archived files will be placed. If nothing is selected the files will be created in the folder where NSLog is located.

Close Stike Limit - The distance that defines which strikes will be counted as "Close".

High Activity Threshold - The number of "Close" strikes per minute that will trigger the High Activity Uploads.

X Minute Strike Recording - Allows you to select the interval for the second set of counters. This can be anything from 2 to 60 minutes.

Processor Priority - Allows you to increase the priority that your processor gives NSLog. In testing, using realtime has resulted in an over 50% reduction of missed events during medium and high activity storms. If you notice any sluggishness of your mouse cursor or any other issues, you should leave this in normal. I haven't noticed any problems with slow downs or with any of the many weather programs that are running on my computer.

Gauge & Strike Colors - Click on any of the boxes to change to your desired color.

Ftp To Use - You have a choice between an internal routine managed within NSLog or using the external nsFtp "fire & forget" program. See the Troubleshooting Uploads tab for more information on both of these. The new internal FTP is highly recommended.

Log FTP Transfers - Will create a log with information about each of the transfers. Use this only when necessary because the log file can grow very large in a hurry. If you're using internal FTP the log will be archived at midnight in the same place that you have designated for NSRaw.txt archiving, and will have the date appended to the file name.

Verbose - (Internal Only) If this is checked it will log all of the communications with the server. Each upload will be 25 lines in the log rather than the normal 6 essential ones, but it might be helpful for troubleshooting your setup or if you're just curious.

Connect Timeout - (Internal Only) You can select how long the ftp will wait before it quits on a connect. 5-6 seconds should be fine and most connects will be nearly instantaneous.

Transfered Today - (Internal Only) Shows how much you've uploaded and if logging is on, this information will be appended to the log when it is archived.

Realtime Tab

Create Realtime File - Allows you to select whether this file is created or not.

Realtime File Folder - Default is in the folder that contains the NSLog.exe program, but the file can be placed elsewhere. Do not select the \nslupload\ folder.

FTP The Realtime File - Allows you to select whether the file will be uploaded or not.

Normal Rate - The rate at which the file is uploaded when High Activity is not sensed.

Accelerated Rate - The rate at which the file is uploaded when High Activity is sensed as determined by the close strike configuration on the general tab.

Login - Standard.

Server - In this example I'm uploading to the /web/ folder on my site which happens to be the root.

Password- Standard.

Passive - Passive if checked, active if not.

File Rename - The file is uploaded with a different file name and then renamed once the transfer is complete. It is recommended to use this if your server allows.

Port - The standard is 21, but you can change to something different if you need to.

Clone Button - While NSLog is designed to allow you to upload files to different servers and different folders, many users will be sending their files to the same place. This button copies the FTP info as entered on the Realtime tab to all the others to make initial setup a bit easier.

Strikes Tab - Just like the Realtime Tab except:

Normal & Accelerated Rate Minutes of Strikes  - Allows you to select how minutes of data are stored in NSStrikes.txt. The range is from 1 to 1440 (24 hours).  The NSStrikes.txt file can get quite large during high activity, so it's probably a good idea to have a reduced number of strikes in the file when those conditions are present.

Storms Tab - Just like the Realtime Tab except:

Create Storms File - NSStorms.txt will be created in the Storms File Folder and will contain all storms that are active at the time. Once storms expire they are moved to NSOldStorms.txt.

Archive Daily - The day's storms that are in NSOldStorms.txt will be archived in the folder that you've selected on the General tab, and will be named yyyymmddNSStorms.txt. If this is not selected NSOldStorms.txt will continue to grow and will have to be moved/renamed/deleted manually.  This allows the creation of monthly or annual storm files if those are desired.

Trac Release Time - Set this the same or longer than you've set in NexStorm. If a storm has no activity for this amount of time, it is moved to the NSOldStorms.txt file.

Minimum Storm Duration - Storms shorter than this are not saved to the NSOldStorms.txt file.

Location of TRACReport.txt - The folder that you've configured NexStorm to create this file in. If in doubt, look in NexStorm's Data Export configuration screen.

Graph Tab - Just like the Realtime Tab except:

Log Interval - Allows you to select how often the current strike data is written to the log file.

Max Lines in Log - Limits the size of the file so it doesn't eat up too much space.

Upload 1 - 5 Tabs

These work just like the Realtime and Graph file uploads, and allow you to upload up to 5 additional files which can be sent to different locations. File times are checked on these files and they will not be uploaded unless they are newer. If you select 0 Minutes in the Accelerated Rate box, NSLog will check your Upload 1-5's every 15 seconds when in High Activity mode. You'll want to select Binary on these if your file is a graphic or other binary file format.

Filters Tab

This tab is not applicable for most users, but will be invaluable to those who have chronic problems with artificial strikes from sources such as an electric fence, malfunctioning street light or a neighbor with a welder.

Two areas can be defined by setting the "right and left" bearings and the "long and short" distances of the problem(s).  Strikes and storms that NSLog sees within these boundaries are then totally disregarded and excluded from all NSLog proveded data. If using this feature, you may want to consider using NSDisplay to show your data on the web because it has now been tailored to do just that.

Troubleshooting Uploads

If using nsFtp the upload times on the NSLog main screen indicate when NSLog last asked nsFtp to upload the files. Because nsFtp is a "Fire and Forget" program, NSLog doesn't know whether the uploads were successful or not. If using internal FTP each upload will start with the word "Upload", then show the number of bytes as they are transfered, and finally the time that it finishes. If you don't get a good connect it'll show the word "Failed" and it's time to check your server and internet connection.

When NSLog determines it is time for an upload, it checks the file time of the target file and compares that with the most recently uploaded copy of that file. If the file is newer, a copy is placed in the nslupload folder and FTP is asked to upload it. By observing the file times in the nslupload folder, you can determine how often each of the files is being created and uploaded. The nslupload folder is created as a hidden folder within the same folder that contains NSLog.exe. In order to see it, you may have to enable "Show hidden files and folders" in Windows Explorer. Don't forget that files are only uploaded when there's a newer one available, so many times NSStrikes.txt looks pretty stagnant on a slow day.

Should you notice that a file is not being updated in the nslupload folder, First make sure that all of the server information is correct on the tab for that particular file. Also check that the file is actually being created in the location that you have specified.

If all of the above checks out, you may try enabling ftp logging on the Configuration-General tab. This can reveal several possible errors. Some servers cannot handle 3-5 second uploads of the Realtime.txt file, but will handle 10-15 seconds just fine. Be sure to turn ftp logging off when you are done because the file can grow quite large in a short period of time. Also, you may delete the log file at any time and it will be re-created by NSLog when necessary.

Internal FTP

Ftp logging looks different than previously with nsFtp so I'll explain some of the details. Here's a typical transfer:

Uploading nslupload\NSRealtime.txt at 06:40:55:437 2010/12/29 (1 in queue)
Connected 06:40:55:562 (00 sec 125 ms)
CWD /public_html/lightning
Opening ASCII mode data connection for NSRealtime.txt.nsl
152bytes received/sent in 344 milliseconds
Disconnecting 06:40:56:468 (01 sec 31 ms)

On the first line, the queue info within parenthesis will only show up when there are other ftp jobs waiting. This can be due to:

1. Uploading too fast for your server and/or internet connection in which case you'll see it a lot.
2. Either your server or internet connection are down or "flakey" which can happen occasionally.

When this happens you should see that the number increase when we send the larger files, then decreases as we catch up. Seeing these entries isn't bad, it just means the queue is doing it's job. But if it's extreme, it might mean that you're updating too frequently for your server or internet connection. If your internet connection or server go down, the queue will be limited to a maximum of 7 so that things don't too backed up.

The times within parentheses on the Connected and Disconnecting lines are elapsed time. If an asterisk shows up next to the connect elapsed time then it's within 1 second of the time you have set in config for the connect timeout. That way you can do a search of the log for "*" to see how often things are getting close for connect times and if you see it a lot, and you're seeing a lot of connect timeout errors consider increasing by a second.

If you have rename selected you'll see that extra ".nsl" at the end of the uploaded filename which goes away after the file is renamed on the server.

Any problems that happen should have the word "error" in the entry so you can search for that. If you're seeing connect timeout errors but they're only every hour or two, you're probably ok because that seems to be the norm on the three servers I tested with.

The way I review the ftp log is to select View-View FTP Log then in notepad search for

1. "error" to see if there were any premature disconnects or other problems.
2. "*" to see if there are a lot of slow conects or if I need to increase my connect timeout.
3. "queue" to see if I've had any of those buildups, but you would probably have already seen those when searching for the above.

Then I'll do a "File-Save As" if I want to keep a copy for something. If I want to clean out the file and start fresh I'll turn off ftp logging then I'll do "Edit-Select All-Delete" then "File-Save" and then turn the logging back on. Try not to save the cleaned out file when an ftp session writing to the log or it "most likely" will lock the NSFtpLog.txt file up. If this happens you will have to shut NSLog down and restart it to get the logging working again.

The ftp logging has a verbose! mode where you'll see all of the communications with the server. Each upload will be 25 lines in the log rather than the normal 6 essential ones, but it might be helpful for troubleshooting your setup or if you're just curious.

If logging is on at midnight, the day's log will be archived in the same place that you have set for NSRaw.txt archiving. The amount of data sent that day will also be appended to the end of it.

If you notice problems, please let me know and send me the log if you had that turned on. A screenshot would be helpful too. If you have some decent lightning activity please let me know how the NSLog counts compare with those in NexStorm.

nsFtp

Some users find that nsFtp will occasionally hang and they find several instances remaining in Task Manager. Often this can be cured by reducing the frequency of NSRealtime uploads. To confirm that an instance is truly hung, configure Task Manager to show the process id (PID) by going View- Select Columns and then tick PID (Process Identifier).

If the same PID remains for more than a minute or so, nsFtp is hung and you can delete that instance by right clicking and selecting End Process. This has been a tough one to find and fix. Sometimes it seems to be an issue with the server, sometimes the ISP and sometimes the uploading computer. Awhile ago I did some troubleshooting with Tom in North Carolina and I could upload to his server just fine at 3 second intervals but he could not. He ended up slowing his uploads down to 10 seconds and things worked much better for him. It has also been noted that some antivirus programs can at times cause a locked file in the nslupload folder, so if you experience that set your antivirus program to exclude the entire NSLog folder from scans and see if that improves things. If you are experiencing extreme problems with this and wish to work with me on it, please drop me an EMail (contact info below) and we'll see what we can learn.

Uploading NexStorm Files

You may choose to have NSLog upload your NexStorm and Wasp2 images and data if you want to let NSLog control the rate of the uploads. The following settings will allow creation of the files on your hard disk without NexStorm or Wasp2 uploading them.

    

NSDisplay 1

May 22, 2011 Version

This venerable old script, originally created by Henkka Roblom back in 2008 before there was an NSLog, has been recently updated to use the NSStorms-include and nslmap scripts below. All display and updates are accomplished using NSLog provided data with no other uploaded files necessary. Since NSLog only uploads files when there have been changes, bandwidth use is greatly reduced.

The use of nslmaps provides some unique auto and manual zoom capabilities that many will like, and can be configured to look exactly like your NexStorm generated images.  NSStorms provides a display of the wealth of storm data that NSLog now provides.

For those who are using NSLog's ability to filter out unwanted strikes (such as from an electric fence) this is the perfect way to accurately show the results. This script can be seen in action on my NSDisplay page and can be downloaded here. View the readme file to learn more.

Radar Overlays for NSDisplay 1

Given a little time and some money, it is possible to have radar overlays that work with NSDisplay (see mine, link above). Take a look at the readme file to see if this is something that interests you. If so, you can download the essentials here.

NSStorms-include.php

This script uses NSLog's NSStorms.txt and NSOldStorms.txt files to create unformatted tables of storm data. It is meant to be included in another page which contains style information that controls how the table is displayed. If called with no parameters the table will contain current storm data, and if called with the "expired" parameter it will display previous storms of the day. This script can be seen in action on my NSDisplay page (if there's any storm action) and can be downloaded here.

nslmap.php

This script uses NSLog's NSStrikes.txt and NSStorms.txt files to place strike and storm symbols on "clean copies" of NexStorm's maps. It is quite configurable for symbology and supports zooming to different ranges based on current lightning activity. A nice feature is that the zoom can be set to key on distant storms until a certain distance, and then key on the closest. There is also a manual zoom option available.

With this script it isn't necessary to upload nexstorm.png periodically all day long if you don't wish to.  It can also be used with maps from a source other than NexStorm as long as your location is exactly in the center of the map and that you know the distance from edge-to-edge for each map. This script turned out to be much more accurate than I expected, can be seen in action on my NSDisplay page and can be downloaded here.

nslog2grlevel.php

GRLevel3 owners may be interested in trying this out. Tom Chaplin from Carterlake.org originally released this code in the GRLevelX owners forum in 2006. This version has been modified to read the NSStrikes.txt file and display your lightning data in GRLevel3 just as his original script did with the Weather Display produced data. The .zip file contains the script that creates the placefile, an icon file and the Xara Xtreme file that created the .png icon file. You can download it here. The placefile produced can be seen here and to see it in action, add that same URL to your placefile manager and select either LaCrosse (ARX) or one of the other radars in the area.

nslog2kmz.php

Once again leveraging Tom's code above, this script produces a .kmz file that Google Earth can read. Users need to select Add-Network Link in Google Earth and then paste http://yoursite.org/NSStrikes.kml in the Link field and give it an appropriate name. Nothing else is required of them. NSStrikes.kml creates a network link that will periodically call nslog2kmz.php which then delivers the compressed data to Google Earth for display. You can download it here.

I've begun using the Google Earth view of my lightning to better calibrate my system and am very happy with the information that it provides.  It has a great ruler to measure distances with, but I found myself wishing I had range rings that matched the distances in segmented ranging.  So, I figured out a way to put together a .kml file to do that and later added another to displays the radials. The package can be downloaded here.

NSGMap.php

Mike Stelmat has released a system of PHP and javascript named NSGMap which reads the NSLog NSStrikes.txt file and creates a .kml file that places your lightning strikes on a Google Map of your area. There are many options such as update rate, range rings and overlays of weather Radar images. You can see it operate on Mike's site at http://weather.stormctr.com/bocafalls/NSGMap/NSGMap.php.  Current status of the project can be found at http://forum.astrogenic.com/viewtopic.php?t=694 and the script can be downloaded from Mike's Site.

nsFtp.exe

This small little FTP program was developed to support NSLog.exe because there didn't seem to be any free command line programs that would support both long filenames and the server rename function. If you wish, this program can be used for uploading other files via batch files. The syntax to use with one space between each parameter is:

nsFtp.exe logit fullpath filename login password server passive binary rename port

logit - Whether to log the transaction or not, T or F. NSFtpLog.txt will be created if T selected.

fullpath - Full path to the file to be uploaded. If the path contains a long filename, or a name with spaces, enclose this in quotes. In fact, quotes shouldn't hurt anything in either case.

filename - The name you want for your file when it's on your server.

login - The login that you use for your server.

password - Your password.

server - Where you want your file.

passive - T for passive and F for active.

binary - T to upload in binary mode and F for ASCII. Text files should be uploaded in ASCII mode and graphics in binary.

rename - T to upload with a ".tmp" extension followed by a quick rename to your desired name. If your server doesn't support rename, use F.

port - Normally 21, but you can change to something else here.

Examples:

nsFtp.exe T C:\NSLog\Testbat\seo.txt newname.txt jmcmurry mypassword jcweather.us/web/ T F T 21

nsFtp.exe F "C:\Documents and Settings\jmcmurry\Desktop\seo.txt" newname.txt jmcmurry mypassword jcweather.us/web/ T F T 21

Strikes JpGraphs

These JpGraph files use the data from NSGraph.txt to plot lightning activity.

NSRaw.txt - Comma Delimited - Updated with each Strike - This is the exact data as read from the Flashgate IPC. This is the same structure used in NSStrikes.txt which is suitable for data needed covering the last xx minutes such as nslog2grlevel.php above.

NSRealtime.txt - Comma Delimited - Updated with each Strike - Suitable for rapid uploads to be used by javascript routines.

NSStorms.txt & NSOldStorms.txt - Comma Delimited - Storm Information for Upload and an Option to Archive Each Day

NSGraph.txt - Comma Delimited - User Selectable Log Interval and File Length - Suitable for JpGraph, GraphWeather and other charts.