Over the Bridge

I've started two new projects! One of them meshes with my existing stuff; the other is kinda unrelated. Both are programming related.

libliquid

I've started writing a C++ liquid parser/renderer, with an eye on embeddability, performance, and sandboxability.

The repository is hosted on github here.

It's still in the early stages, so likely has more than a few memory bugs, but I've got bindings up and running for C, and Ruby, Perl (both using the C bindings). I've also strapped it to cmake, so there's a semi-legitimate build process involved (though my development builds are still built through a hand-written makefile.

I'm planning to embed it in nginx-xapian, my other open source C/C++ module. So we'll see how it goes. Early indicators are good, though! I've already got it more feature-ful in terms of syntax than Shopify's existing liquid implementation (i.e. we support parentheses and actual operators), if you want them; though you can always implement the strict Shopify version with a single call.

Another cool thing about it is that it allows for multiple dialects, like with the perl liquid module, and almost all the functionality of the language is tied to the dialect, rather than the core implementation. The only tag that exists in the core is {% raw %} (as that's a lexing hint); everything else can be modified by anything using the library; allowing you to very easily add new tags, operators, filters, and other constructs as you load the thing up; though it does come with a fairly comprehensive standard liquid implementation, that includes all the normal greatest hits like {% for %}, {% if %}, and the like; the only thing it doesn't have is stuff that would require external libraries to properly implement, like JSON serializing, and whatnot; but that will eventually be provided with examples.

Performance is good too; I haven't really even done an optimization pass, and it already runs 10x faster than Shopify liquid does, even when called from Ruby with near-identical calls (both parsing and rendering). I think that, combined with the embeddability is going to make this a general win for most projets I want to throw it into; some of which will actually be work-related (though I'll only work on the library in my spare time).

I'm aiming to keep the full implementation under 5KLOC, and there's definitely room for improvement on the way it was done; but I'm hoping, after a few polish passes, it'll actually work nicely both at work, and in hobby projects.

Pre-Confederation Ornithology

My dad has always been majorly involved in birdwatching; being in the executive of a number of non-profit birding organizations around Montreal. Over the years, he's aggregated and transcribed a ton of early Canadain material on ornithology, and has finally decided to publish some of it online. I've helped him create a website for this purpose over here.

Looks familiar, doesn't it?

In addition to helping my dad, also helped me refine the static-site generation framework I wrote for fun, that builds this site. So it's a win-win all around.

So I've added in a search bar to the site.

This is a bit more involved than it sounds. One of the things that I really want to keep this site, is static, without any javascript. Or rather, static as much as possible, self-contained with a really simple, cheap, and cost-effective webserver (nginx) serving up the content.

The issue with this is, you can't really do anything dynamic (based on something the user types in, for example), you're limited purely to static stuff, unless you want to write some C and write an nginx module.

So I did. And here's the github repo which hosts the code I'm using to run the search: https://github.com/adamharrison/nginx-xapian.

I know, I know, it's currently riddled with bugs, and is frankly, unsafe. I shouldn't be using it on this live site. It's relatively safe because this machine doesn't really contain anything of value, and is just a VM in Amazon's cloud, which I can simply kill at any time, without losing anything, but still.

That's what I've been working on for the past week or two in my spare time, an hour here, an hour there, in between writing. In the end, I'm actually pretty pleased with it; it's pretty quick, and while it could be a lot more efficient, and there's a lot of work to do in terms of making it update its indices dynamcially based on files updating, and whatnot, but it seems to get the job mostly done.

So, search is up, with no javascript, and no script-based back-end. Good stuff.

I've added in a project sections up above to showcase some of the stuff I'm doing, and its current progress. So far, I've just got my writing stuff in there: some details about my first book, The First Estate (probable title), and a small stub about my current project, Eldar's End (tentative title). Currently, it's just a static thing, but I'll probably be working on it going forward to automatically pull in some of my stats.

For example, I have the blog system set up in such a way that I can easily just add little snippets of script if I need to (without actually having a huge whole framework involved in the actual project); I'm already using it to pull in the "What I'm Reading" and "What I'm Playing" from a remote site that tracks my literature/media consumption.

I'm thinking I can include either a progress bar, or a simple word count setting. Progress bar probably makes more sense, because I already have a target amount of words I want to hit for each book I write (100,000), which is a nice blend of "long-enough" and "not too long". All in all, that works out to about a 350-400 page book, which is about stadnard for a novel. I was able to hit that target within 10% for my first book (I ended up writing 110,000 to wrap everything up), and I'm going to try to hit it for my second book as well. So a little progress bar would probably be nice.

I'll probably have it up by next week.

In general, you'd think gauging the performance of a piece of code is easy, right?

Well, I was under that same impression. Even at work, whenever I had to figure out whether something was running faster, my frist instinct was to test it a couple times on a few input sets, and if it sort of felt right, then, well, that was good enough.

I saw fascinating talk over the weekend, "Performance Matters" by Emery Berger. (Linked Below)

Basically the gist of it is that there are a number of things which can randomize performance of native applications (to say nothing about interpreted applications), and that it's relatively hard to determine whether or not any particular change you make (beyond ones that are extremely obvious that make things an order of magnitude faster or slower) is actually affecting the software in a positive or negative way. Small gains in performance, can actually be due to things like memory layout, rather than any particular change you've made.

Essentially, Berger and his team have developed a tool that randomizes the memory layout of the compiled application (amongst other things), that allow you to perform a bunch of runs and get a reasonably accurate distribution of performance measures that will help you figure out whether or not you've actually improved your software.

One of the most imporatnt points he drives home is that, even though you now do have data that accounts for memory layout changes, you can't simply look at it, and do a simple linear comparaison between previous results, and say it's faster. He refers to this as "eyeball statistics". Basically, he makes the point that you want to a take statistical approach to this situation, and make statements like "within a 95% confidence interval, what is the probability that the speedup we're seeing here is by chance?". He does assert that the runs are normally distributed (he does say there are reasons for this, but to my memory doesn't go into them), and so thus, we can treat any run data we have as a normal distribution and perform some easy stats on it to get our answer to that question.

Anyway, by far and away the most interesting talk I've seen in the past couple years. Give it a listen, if you're interested in optimization with any sort of native application.

So, as of December last year, ish, I started developing a desktop game in C++.

This all started as a throwback to development I used to do back in the day. I really like game programming, especially if I get to write the engine. So even though it's obviously going to take more time if I take that route, as opposed to just pulling an engine off the shelf.

In the end, I'm basically compromising. I'm pulling all the individual components of the engine (i.e. scripting language (Lua), GUI toolkit (nanogui-sdl), windowing stuff (SDL)) off the shell, and just gluing them together in the way that I think is appropriate. So far, it's worked quite well, and I have a super basic game up and running.

I'm a huge fan of simulation games; in particular, I love Rimworld. The idea of running a colony, where each individual has meaningful interactions was always my ideal. Rather than dealing with a formless blob of "population" that has no characteristics whatsoever. Quite frankly, I'm usually more interested in games, the fewer PCs are around. I always play any party CRPG solo, if possible (e.g. Fallout with no companions; Age of Decadence as a single character, etc..).

And so, it got me to thinking. Why isn't there a colony-sim-type-game where you only have a couple pawns? I mean, it's less of a colony sim, and more of an RPG at that point. But what if you did make it interesting enough to run through things with just a few pawns? And if that's the case, and it is more like an RPG, why not throw in more RPG elements, and make some sort of hybrid?

Now, I'm sure this already exists. Most ideas have already been executed. But I don't know of any. (Maybe Dwarf Fortress? I've never played it.) So, in any event, I decided to use this as a base for my game. I'm taking inspiration from the colony side of things from Rimworld, and the RPG side of things from ADOM. I've decided to go for fantasy, over sci-fi, as I tend to like that genre more than the other.

The goal of the game, is to progress to the top/bottom of a dungeon, which your primary colony sits outside of. I'm unsure of the flavour of this at the moment, but could be something like "Climbing the World Tree", or somesuch. Your town will occasionally get raided by dungeon denizens, and as you dungeon crawl up the tree/down the dungeon, you'll encounter stronger and stronger enemies, yet get greater, and greater rewards.

My guiding philosophy for desinging this thing is as follows:

Truly Meaningful Modifiers

The issue I have with a lot of colony sims, is that when you get a modifier for a pawn, chances are, it's basically meaningless. Sure, +10% workspeed is cool and all, but in the end, can you really say it affects how you actually treat that pawn? -25% learning speed? One thing that ADOM did quite well, was that every significant bonus you got from an artifact in that game was incredibly meaningful. Almost every single non-guarnateed artifact you got had a good chance of fundamentaly changing how you played your character.

Ideally, in this game, I'd like to have that. So if you have traits, on characters, these traits shouldn't be like +10%, or +25%, but something on the order of +250%, or higher. Sounds weird for balance purposes, or even for any sort of storyline purposes, but I just can't help but thinking this would be a hell of a lot more interesting than otherwise. Ditto items; it shouldn't be a differential of "this item is 10% better at this job than the other", it shoud be an order of magnitude different if you're going to get the player to care about it.

The Problem with Caravans and Multiple Bases

In Rimworld, one of the issues I had was the caravan system. I basically never use them. You rarely need to, and they don't give me a great vibe. I don't like the idea of striking out, away from the support of the main colony, when almost all members of the colony are pretty integral to its functioning, and if something goes wrong (like a major raid), you can be pretty screwed. That said, I do like the dungeon crawling experience in ADOM and other roguelikes; basically delving deeper, and deeper in search of better loot, and more dangerous monsters. Sounds almost like a contradiction.

There's also not much reason to establish new colonies, as that's just more territory to defend, and there's not really a compelling reason to do so. Even though it's fun to build new bases!

The way I intend to resolve this dichotomy of not wanting to venture out with key personnel, yet still retain the need for a roguelike dungeon-delve will be the ability to establish outposts of your main colony that allow for instantaneous travel between. In this way, you can you still venture out, but still come back relatively easily, in case something happens. Kind of like the town portal mechanic in Diablo.

I intend to solve the problem of having no reason to build smaller outpost bases by making them very easy to integrate into your main base, with easy travel between, and have exploitable resources actually entice you to drop a new outpost in the first place. That way, you can go through the whole (hopefully enjoyable) base building mechanic several times, and achieve something meaningful.

Playstyles

Early on, I decided I really wanted to have different viable playstyles. It doesn't matter if they're enforced as hard modifiers; I don't mind that. But the goal would be to create one system that has quantaitive modifiers that are so large, or straight qualitative modifiers that they fundamentally change how you play the game. I decided, that the way to do this was to split things off into races.

So, for your colony, you'll have a choice of picking a race for your colonists which has a significant effect on how your colony runs, but first a bit of background on pawn mechanics.

Background

Every pawn has an age. It'll likely be marked in hours. Age won't really do a whole lot, but after ~4-6 hours, an unmodified pawn will die. Basically, long enough to get attached to an individual pawn, but not so long that there's no turnover. Pawns can breed, and produce children. Every pawn has skills; some of which are mundane, some of which are technological, and some of which are magical. These skills are unique to the pawn, and the pawn levels them up at a given rate by perfroming the tasks that those skills influence. Each pawn has item modeifiers, modified by items that they're wearing/holding. Lastly, all pawns have technological modifiers, which are the same for every pawn, and are given by various technologies that can be researched.

Elves

Elves are the stereotypical tolkien-esque variant. They have the following modifiers:

• -95% Fertility
• -100% Aging Speed (Infinite Lifetimes)
• -100% Technological Skill Learning Speed
• -100% Research Speed
• -100% Disease Vulnerability

A colony of elves is designed to consist of a few pawns. Adventuring parties of elves are usually either 1 or 2 pawns maximum. Designed to be incredibly overpowered individually, yet very individually precious. Designed to to use any technological modifiers; depends purely on individual pawn skill. Meaning that individual pawns are highly specialized and close to irreplacable. Because pawns are immortal, and do not die, except through violence, or privation, and very high skill levels make up for lack of technology. Name generation system will spit out long names with lots of apostrophes. Things are generally manged on an individual level. (Pawn priorities/attire set individually)

Humans

• -75% Magical Skill Learning Speed
• Ability to pass on skills to children that share last names.

Humans are meant to be the vanilla race. They allow for everything, but don't excel at anything. When they produce a child, that child forms part of the lineage of one of the parents. In this way, you can slowly breed particular family lines to specialize in particular skills, while also benefiting from advancing technology. Names will be common western fantasy names; a first and a last name. Children of a union will always bear the name of the mother, or the father, depending on colony settings. Things are generally managed on a class level. (Pawn priorities/attire set based on a preset designation that's shared between a number of pawns)

Dwarves/Gnomes

Haven't decided on which to use. Gnomes makes thematically more sense, but I like the idea of dwarves.

• +400% Aging Speed (~1 Hour Lifetimes)
• +400% Fertility
• -80% Skill Learning Speed
• +100% Technological Reseach speed

Dwarves are meant to be the big bio-blob race. Think drones in factorio. Basically, the idea is that you simply produce a huge amount of pawns, all that are pretty terrible at their jobs, and die quickly, but you just have so many of them that you simply expand to fill your entire agricultural capacity, and continuously advance through technological change alone. They'll have very similar names to encourage not differentinating between them. (e.g. Glim, Glom, Gloin, Gluin, Glon, Glan, Glin, etc..). Things are generally managed at a societal level (Pawn priorities/attire set universally, among all pawns).

I figure I can put that together fairly easily just using those numeric modifiers, but they'd end up being super different in terms of playstyle.

Timeline

Anyway, since I started this thing in December, I've decided to take things very incrementally; putting in a bit of time here and there, based on very clearly defined goals. Here's my list thus far:

• Get a basic tile grid displaying, which loads assets from a basic management system. DONE 2019-12-10
• Throw in rocks/trees onto the map. DONE 2019-12-17
• Throw in multiple different tile types at random. DONE 2019-12-22
• Throw in a pawn, which sits there. DONE 2019-12-22
• Throw in a pawn which walks from one side of the screen to the other, ignoring obstacles. DONE 2019-12-24
• Creation of first true map, via Lua, by manually adding Actors. DONE 2019-01-07
• Throw in a pawn which interprets rocks and trees as obstacles, and routes around them with A*. DONE 2020-01-07
• Allow selection of a pawn, with right clicking allowing the user to choose a destination for the pawn, and then have the pawn move there. DONE 2020-01-12
• Creation of UI. Choose a GUI toolkit library, and draw basic pawn selection UI that displays name of pawn. Also, create contextual click menu, that can be triggered on command. DONE 2020-02-01
• Allow dragging creation of zones. DONE 2020-02-04
• Create a basic lua console. DONE 2020-02-16
• Creation of the item of "Potato Seed", and place it on the ground. DONE 2020-03-02
• Create an inventory for the player, and have them be able to pick up the item on the ground. DONE 2020-03-02
• Allow for the potato seed to be picked up, and placed into the pawn's inventory. DONE 2020-03-02
• Creation of farming skill associated to pawn. Pawn can be given a planting zone. DONE 2020-03-02
• The pawn can be manually given orders to plant in Planting zones. This creates another actor (growing seed), decrementing the appropriate seeds from the inventory. DONE 2020-03-03
• Have growing seed have its own informational dialog, which displays its name, and how long it'll take to grow. Make this into a general dialog that can show details about any actor. DONE 2020-03-10
• Have growing seed actually grow from small, to large, to ready. DONE 2020-03-10
• Once ready, have user be able to give an order to a pawn to harvest the plant. This should create an item in the pawn's inventory. DONE 2020-03-10
• Allow for the display of a pawn's inventory.
• Allow for a save state. Should save all actors on the map, and all tile states. Loading should, well, load this, and the entire Lua state. The lua state is described by every single entry in the registry table.

So, that's where I'm at right now. I still have to figure out art and stuff; I'm just using placeholders for now. But I figure I'll worry about art later, after I've got the core gameplay loop at least up and running. The plan is for the whole thing to get a very basic alpha up in about a year. We'll see if that's feasible. Ironically, COVID-19, with its disruption to my daily schedule has made me less produtive; rather than more, even though I'm stuck at home most of the time. I'll post occasional updates here about how I'm doing on this little project.