Accessing total "billable" traffic from Modem Router

I realize I may be pushing my luck with this question, but I am looking for some initial guidance on

  • the possibility of making this happen, and
  • where to go look for that guidance.

My modem is a Hitron CGN3 Wireless Modem Router.

Is there any way, other than performing a daily "web-page scraping", to obtain my own ongoing cumulative data usage report? The ISP's web page is notoriously unreliable, sometimes showing the same number for days.

The Modem is DOCSIS 3.0 .

Does anyone know of a way I could access the WAN-side "traffic" or "ports" to probe and obtain that cumulative total GB for a specified period?

Any possible "IP-carried" commands that would trigger that modem's admin port (192.168.0.1) to respond to my queries about cumulative traffic/logs usually reserved for the ISP ?

If web-scraping is the only possible method (sample page),

can anyone provide a reference to a "simple" guide to web-scraping, allowing me to harvest the data from the fields/boxes

  • WAN Receiving, and
  • WAN Sending

... and to do that, somehow, by performing a "scripted login session" to the Modem's admin account to access those pages, and locate references to the following (typical) content:

<div class="span6">
<table class="table table-striped table-bordered">
<tbody><tr><td width="35%">WAN IP Address</td><td>174.116.29.27, </td></tr>
<tr><td>WAN Receiving</td><td>19.08G Bytes</td></tr>
<tr><td>WAN Sending</td><td>1.29G Bytes</td></tr>
<tr><td>Private LAN IP Address</td><td>192.168.0.1/24</td></tr>
<tr><td>LAN Receiving</td><td>1.29G Bytes</td></tr>
<tr><td>LAN Sending</td><td>19.53G Bytes</td></tr>
<tr><td>WAN Up Time</td><td>009 days 05h:36m:58s</td></tr>
</tbody></table>
</div>

I don't know if this will give further insights as to what is/isn't possible:

Have you looked at vnstat ? ... it can monitor your WAN ip

https://openwrt.org/docs/guide-user/services/network_monitoring/vnstat

1 Like

In theory such the routers usually allow SSH access to their Unix/Linux - like CLI. SSH access to a device can be easily automated in Python with Paramiko library.

https://www.linode.com/docs/guides/use-paramiko-python-to-ssh-into-a-server/

Another option is to issue ssh -l <user> <destination> <remote command> Entering ssh password can be automated like this: sshpass -p<password> ssh ...
Unfortunately, ISPs usually reserve SSH access for themselves and prohibit it for the end users. I'd suggest that you find out what is your situation.

Thus, it well might be that web-scraping is the only feasible method to extract data from web modem/router. When you know your URL you can try to extract page in question using curl command line utility. curl supports authentication. E.g.

Or you can write scraping script in Python.

Good luck!

1 Like

Thank you, Pavlos.

apt-get doesn't find "luci-app-vnstat". Is there an alternate name?

Also, my computer is not directly attached to the WAN. The ISP's modem stands between the two. I get my IP address using:

inxi -Nni -c0 | grep 'WAN IP:' | cut -f2 -d\: | cut -c2-

Full output from "inxi -Nni -c0" is as follows:

inxi -Nni -c0
Network:
  Device-1: Qualcomm Atheros AR8121/AR8113/AR8114 Gigabit or Fast Ethernet
    driver: ATL1E
  IF: enp2s0 state: up speed: 1000 Mbps duplex: full mac: {myMacAddr}
  IP v4: 192.168.0.10/24 type: dynamic scope: global
  WAN IP: {myWanIpAddr}

Given that, how do I update /etc/vnstat to implement the step equivalent to:

Interface "pppoe-wan"

Thank you, Eugene.

I just can't seem to get curl to work for me. Here is the script I used:

NET__Modem_GetReport_Status.sh :

#!/bin/bash

###  REF:  https://blog.apify.com/basic-auth-in-curl/
#	curl -u username:password [<https://example-protected-resource.com>](<https://example-protected-resource.com/>)

adminUser=cusadmin
adminPwd=adminPasswordNotRevealed

modemLoginPage="https://192.168.0.1/login.html"
modemIndexPage="https://192.168.0.1/index.html"
modemDataURL="status_system/m/1/s/1"

credentials="${adminUser}:${adminPwd}"
token=$( echo "${adminUser}:${adminPwd}" | base64 )

###
###	This form of command failed
###
#curl --insecure --user ${credentials} "${modemLoginPage}#${modemDataURL}"
#curl --insecure --user ${credentials} "${modemLoginPage}"



###
###	This form of command failed
###
#curl --insecure -H "Authorization: Basic ${token}" "${modemIndexPage}#${modemDataURL}"
curl --insecure -H "Authorization: Basic ${token}" "${modemLoginPage}"

Not sure, but I thing the fact that the URL for the top Index page is not the same as the URL for the Login page defeats the whole attempt at accessing the "data" side of the Router. :frowning:

Login page (.../login.html):

Status Page (.../index.html#...):

The Status page comes up as default page after a successful login, but my attempts all fail.

Full "log" from attempt:

<!DOCTYPE html>
<html lang='en'>
<head>
<meta charset="utf-8">
<title></title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="">
<meta name="author" content="">

<!-- Le styles -->
<link href="css/bootstrap.css" rel="stylesheet">
<link href="css/bootstrap-responsive.css" rel="stylesheet">
<link href="css/login.css" rel="stylesheet">
<!-- Le HTML5 shim, for IE6-8 support of HTML5 elements -->
<!--[if lt IE 9]>
<script src="lib/html5shim.js"></script>
<![endif]-->

<!-- Le fav and touch icons -->
<link rel="shortcut icon" href="ico/favicon.ico">
<link rel="apple-touch-icon-precomposed" sizes="144x144" href="ico/apple-touch-icon-144-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="114x114" href="ico/apple-touch-icon-114-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="ico/apple-touch-icon-72-precomposed.png">
<link rel="apple-touch-icon-precomposed" href="ico/apple-touch-icon-57-precomposed.png">
<script type="text/javascript" src="lib/jquery-1.8.3.min.js" ></script>
<script type="text/javascript" src="lib/jquery.hitronext.js"></script>
<script type="text/javascript" src="lib/dict.js"></script>
<script type="text/javascript" src="lib/bootstrap.min.js"></script>
<script type="text/javascript" language="javascript">
var skip_Wizard;
var model_name;
var user_type;
var DefFirstUrl;
var DefNormalUrl;

function userlogin() {
$('.alert').hide();
$('#btnLogin').button('loading');
if ($('#user_login').val() == '') {
$('#cg_username').addClass("error");
$('#btnLogin').button('reset');
return false;
}
else {
$('#cg_username').removeClass("error");
}

if ($('#user_password').val() == '') {
$('#cg_password').addClass("error");
$('#btnLogin').button('reset');
return false;
}
else {
$('#cg_password').removeClass("error");
}
//start submit the form
if ($("#user_remember_me").attr("checked") == "checked") {
setCookie("userName", $("#user_login").val(), 24 * 30, "/");
setCookie("password", $("#user_password").val(), 24 * 30, "/");
} else {
deleteCookie("userName", "/");
deleteCookie("password", "/");
}

$.post(
"goform/login",

{
"user": $('#user_login').val(),
"pws": $('#user_password').val()
},

function (result) {
$('#btnLogin').button('reset');
if (result.indexOf('success') < 0) {
$('#Error_Unknown').html($.tag_get(result));
$(".alert").fadeIn();
} else {
$.getJSON("data/user_type.asp", function (json) {
user_type = json.UserType;
DefFirstUrl = json.DefFirstUrl;
DefNormalUrl = json.DefNormalUrl;
});
$('#login').fadeOut(function () {
if (skip_Wizard == "1")
window.location = 'index.html#' + DefNormalUrl;
else
window.location = 'index.html#' + DefFirstUrl; /* first login the page */
});
}
}
)
}
$(document).ready(function () {
$.hitron.languages.lang_init();

//$('title').html($.tag_get('Login_Page_Title'));
$('#btnLogin').attr('data-loading-text', $.tag_get('Login'));
//Load title and vendor name.
$('#user_login').attr("placeholder", $.tag_get('Username_Placeholder'));
$('#user_password').attr("placeholder", $.tag_get('Password_Placeholder'));

$.getJSON("data/system_model.asp", function (json) {
document.title = json.modelName + ' ' + $.tag_get('Login_Title');
$('#Login_Title').html(json.modelName + "<span> " + $.tag_get('Login') + "</span>");
if ($.hitron.languages.lang_current == 'fr_CA') {
$('#Login_Title').css('font-size', '16px');
if (json.modelName == "CGN3ACSMR") {
$('#Login_Title').css('font-size', '15px');
}
}
$('#login').fadeIn('slow');
skip_Wizard = json.skipWizard;
model_name = json.modelName;
var cookie_model_name = getCookieValue("modelname");
if (cookie_model_name == "") {
setCookie("modelname", cookie_model_name, 24 * 30, "/");
}
setCookie("cur_modelname", model_name, 24 * 30, "/");
});

//Login button function
$('#btnLogin').click(function () {
userlogin();
});
$('#user_password').keydown(function (event) {
if (event.which == 13)
userlogin();
});
//close alert
$('#btnCloseAlert').click(function () {
$(".alert").fadeOut();
});

$('#selLanguage').change(function () {
$.hitron.languages.lang_set($('#selLanguage').val());
});

var userNameValue = getCookieValue("userName");
$("#user_login").val(userNameValue);
var userNameValue = getCookieValue("password");
$("#user_password").val(userNameValue);
if (userNameValue != "") {
$("#user_remember_me").attr("checked", true);
} else {
$("#user_remember_me").attr("checked", false);
}

$.hitron.languages.lang_load();
$('#selLanguage').val($.hitron.languages.lang_current);

var date, year;
date = new Date();
year = date.getFullYear();
$("#copyright").html("&copy " + year + " " + $.tag_get('Copyright'));

});
</script>
</head>


<body>
<div class='container' id='login' style="display:none">
<span class="hide">RedirectLoginIdentify</span><!-- For identfying session timeout event for Backbone.js -->

<div style='width:auto !important;width:410px;max-width:410px;margin:auto;height:auto !important;height:410px;min-height:410px;' id='loginWindow'>
<div class='section section-large' id='login-section'>
<div class='section-header section-header-black' large='large'>

<div style="float:right;">
<img class="logo" src="img/logo_small.png"></img>
</div>
<h3 id="Login_Title"></h3>
</div>
<div class='section-body'>

<div class="row-fluid">
<div class="control-group" style="margin-bottom:0px;" id="cg_username">
<label for="user_login" class="langTag" id="Username" ></label>
<div class="controls"><input class="input-xlarge" id="user_login" name="user[login]" maxlength=128 type="text" />
</div>
</div>
<div class="control-group" style="margin-bottom:0px" id="cg_password">
<label for="user_password" class="langTag" id="Password"></label>
<input class="input-xlarge" id="user_password" name="user[password]" maxlength=128 type="password" />
</div>

<div>
<label class="langTag" id="Language"></label>
<select id="selLanguage" style="width:280px">
<option value="en_US">English</option>
<option class="langTag" id="French" value="fr_CA"></option>
</select>
</div>

</div>
</div>
<div class="section-foot">
<div class='row-fluid'>
<div class='span4'>
<button class="btn btn-primary" name="commit" type="button" value="Login" id="btnLogin">
<span class="langTag" id="Login"></span>
</button>
</div>
<div class='span8' style='padding-top:5px'>
<label class='checkbox' for='user_remember_me'>
<input name="user[remember_me]" type="hidden" value="0" />
<input id="user_remember_me" name="user[remember_me]" type="checkbox" style="margin-top:-1px\9;" value="1" />
<span class="langTag" id="Login_RememberMe"></span>
</label>
</div>
</div>
</div>
</div>
<div class="alert alert-block alert-error fade in" style="display:none" >
<a class="close" href="#" id="btnCloseAlert">&times;</a>
<h4 class="langTag" id="Login_Failed"></h4>
<p><span class="langTag" id="Error_Unknown"></span></p>
</div>
</div>
<div class="footer" id="footer">
<center>
<p id="copyright"></p>
</center>
</div>
</div>
</body>
</html>

Base ... sudo apt install vnstat vnstati

I added the genimages script (Image Generation section) in /etc/cron.hourly/

every hour it runs vnstati and creates 4 images. I made an index.html in my web server to show me those images. I can specify my interface as enp3s0.

My output:

Thank you again, Pavlos.

My interface card is LAN-facing, not WAN-facing.

My interface (equivalent to your enp3s0) is enp2s0 and its IP address, being local is 192.168.0.[2-24] .

Do I enter that line as

Interface enp2s0

(that doesn't see the ISP's WAN side of the modem)
... or ... is there a different specification required?

In the genimages script I have ...

#!/bin/sh
# vnstati image generation script.
# 
 
WWW_D=/var/www/html # output images here
LIB_D=/var/lib/vnstat # db location
BIN=/usr/bin/vnstati  # which vnstati
 
outputs="s h d m"   # what images to generate
 
interface="enp3s0"
 
    for output in $outputs; do
        $BIN -${output} -i $interface -o $WWW_D/vnstat_${interface}_${output}.png
    done
 
exit 1

# REF: https://openwrt.org/docs/guide-user/services/network_monitoring/vnstat
1 Like

Since both my wife and I are sharing the LAN-side of the router, specifying only my interface does not give the aggregate TOTAL GB of Usage, which is what I am trying to get. Also, the LAN-side numbers always seem higher than the WAN-side numbers, by about 2%, which is why I am trying to get at the WAN-side of things.

The DOCSIS modem is 192.168.0.1 and I guess it is integrated and provides wifi to the house, clients 2-24. You cannot upload a script to the modem.

The example provided by the original link was he is using OpenWRT in which you can run scripts on. Thus, you have commulative results for ALL clients 2-24

1 Like

So, you are saying that the vnstat solution is not a workable solution for my context. because my provider's modem categorically does not support OpenWRT. Correct?

That is disappointing. :frowning: But thanks again for having looked at this for me.

You can add a openWRT box 192.168.0.2 next to your modem. In your modem disable wifi, DHCP, DNS, modem will just act as a passthru.

Now, configure openWRT to provide wifi (DHCP, DNS) for clients 2-24.

Then, you can upload the script so that it runs on 192.168.0.2 which will cumulatively will add all interface data from all clients.

basic pictorial

WAN -- modem -- openWRT -- LAN  -- Eric .10
                                -- wife .12
                                -- kido .14
1 Like

The vnstat command supports merging interface data internally using the + syntax, like enp2s0+enp3s0 assuming there is Eric (enp2s0) and wife (enp3s0) or using ip's -i 192.168.0.10+192.168.0.12

From man vnstat

-i, --iface interface
       Select one specific interface and apply actions to only it. For queries, it is possible to merge the information of two or
       more interfaces using the interface1+interface2+...  syntax.
1 Like

Thank you, Pavlos.

I think I need to think on this much more, and dig deeper, before I commit to implementing.

I do very much appreciate that you provided me with all those details. I may not come back on this for at least a week.

Sorry, Eric, can not tell you much. I've verified how my modem's web-interface works and have found out that it happily returns status page code in reply to

curl -u <user>:<password> http://<router's IP>
1 Like

Thank you again, Eugene.

I tried that last method with only the IP address (both https and http) and still got not joy! :frowning:

I heard about the "expect" function to program interractions. I will see if I can make that work for me.

Also, I wonder if I should try using "headless" firefox instead of "curl", in case that would make a difference here.


----- edit -----

Offering this up as FYI only. Not looking for an answer/workaround regarding headless mode.

Scratch the "headless" approach for Firefox. It is always failing (hanging) with the same message, about "no dt" (no desktop???). I do have a Firefox window open on another desktop.

firefox -new-window https://example.com -headless

*** You are running in headless mode.
[GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
^C

Hello, Eric! Have a look at

https://www.zenrows.com/blog/web-scraping-login-python#scrape-sites-requiring-login

which teaches how to find out what data are expected to be sent when 'login' button is pressed. As soon as a payload for post method is deduced, curl can post it

2 Likes