NE Disc App Stats Website
Tue Dec 27, 2022 3:24am

edit: I'm not sure how I rambled on this long....

So, I figure there should be a bit of background on this site I've created. On this actual site, I have an existing report. It's my daily admin report email that gets generated. It has information that I can gather based on querying the database. There's absolutely nothing user specific and it's all just general information. (Ex: number of failed log ins, number of hits, number of subscribers, etc) It's helpful in some situations like the time that I saw I had 50k failed log in attempts in one day. In that case, I was able to track down the offending IP in the logs and block them.

What information that report doesn't have though is any timing information. There's no information how long requests actually take. That's where thsi site comes in.

Thankfully Heroku has a separate log line specific to how long it churns doing the connection as well as my code spins processing the request. (I guess similar to the Tomcat timing logs but more specific.). I decided that this information would be good to know on a historical stand point. My actual production logs are only at most 7 days old if I want to dig so I had to come up with something.

I decided to make a Perl script that scrapes these logs every hour in a cron job for the past hour and the does some fancy regex magic to convert that log line into a hash that can easily be imported into a database. That Perl cron job currently loads up the data into local database as well as I spun up a free Postgres instance online. I figured it has zero user info so it's more or less safe to be remotely hosted. I would have use Heroku but they removed their free stuff.

Querying the database via the CLI was cool and stuff but I'm lazy so why not make 1000x more work by building a web interface to display the DB info all pretty?? Well, that's exactly what I did. Enter this new website.

I created it in Perl using the Mojolicious framework. It's a really nice framework that is surprisingly easy to use. I've used their "Mojo::Lite" options a few times before and it's really great at just throwing together test website/web services in a single Perl script with minimal lines of code. The full blown Mojolicious framework is a bit more in depth but still light weight for the most part.

Anyway, this new site, has more options than I'm going to go into on this post and it's still expanding because I find it fun to develop for and it's somewhat useful.

In it's current state, it shows the offending longest average response time URLs in the first table and then all data for a date range below. I decided to go for the same "DiscApp" style presentation.

Annnnnnnnnnnnnnnyway, I've been babbling on this for a long while now. It's just fun to have another Perl project to work on.

Url to site:

Test username: guest
Test password: guest_password

No guarantee how long I'll keep that u/p open for now that it's posted publicly. If I get any info that the site's chugging data (ie: some bot is spamming) I'll lock it down. That said though, it's a completely different database than the main site so even if it gets spammed, this site won't be taken down with it.


    • Added metrics to the metrics? Sure...? why not. - Erik_, Fri Jan 06 2023 12:23pm
      It turns out Mojolicious has a plugin Mojolicious::Plugin::Status which sort of does the same thing I created for the NeDiscApp site but automatically for apps using the Mojolicious framework. I guess if I wrote the DiscApp site in Perl with Mojolicious, this whole mini project would have been moot.... more
"Forces act when not restrained" - Puckdropper