What and why.
I don’t normally write about Bimbeats since it’s a paid software, and I feel like promoting it on my blog would somehow violate my integrity. At the same time I have been writing about Revit for years, and that sucker sure as hell isn’t free. I don’t know. It just feels weird drumming up the thing that you are profiting from.
The thing is that Bimbeats is built on top of a really cool technology, and it would be a shame if I didn’t write about it. It doesn’t matter if you are a Bimbeats user, and as a matter of fact you don’t have to be to be able to take advantage of these techniques. Whether you are getting your data from Bimbeats, or you setup your own Elastic Stack, it will potentially be useful for you to know how to setup Scripted Fields, and a Watcher to send you Slack messages. Who knows, maybe it will come handy to know this one day.
So, I have mentioned Elastic Stack already. It’s the technology that Bimbeats is built on. Couple of key components of that technology stack are the database, also called the ElasticSearch, and Kibana which is the web-based front end for managing your database, but also to create visualizations. Bimbeats is the data piece here. It collects data from popular AEC software like Revit, Rhino , Dynamo, Grasshopper to name a few. The cool aspect of it is that we built it to be real-time. What’s the benefit of that you might ask?
How do I know if you are working in a Central File?
Well, getting real-time data from Revit can allow you to automate certain responses to, let’s say, some unwanted behavior. Take for example the idea that you can know in real-time when someone accesses the Central File. Be it by accident, unknowingly, or because Revit sometimes unchecks that “Create New Local” button for you. I know, Revit can be a real asshole. It doesn’t happen often, but in Central Files with a handful of users, it can happen when model is busy, and that option is not available for a brief moment. The solution there is to just wait for a few minutes, and try again. However, newer users might not know about these things, and it’s not easy to re-create this state to educate people on what to do. So let’s just assume that someone did accidentally get into the Central File. What’s the big deal?
It’s not always a big deal, but on a big project, with lots of people synching changes to the Central File all the time, if you spend a few hours in that Central File, and realize that you cannot save your work, that might be a bit of a downer. There is more info about that state, and why it can be an issue in Steve Stafford’s post here. For the most part, the likely outcome of this will be that someone will lose a few hours worth of work. Time is money, so let’s talk about how to prevent that, and how Bimbeats can help with that.
Bimbeats pulls data from the Revit file on an “Open” event. That’s when you literally hit the File > Open button, and Revit is loading it for you. The second that file opens, Bimbeats sends off a notification along with some data to your ElasticSearch database. We can see that in Kibana using the Discovery tool.
- Set the filter for “action.name : “Open”” so that you only show Open events data.
- Set the filter for the time range that you are interested in.
- Set the filter for source of data. In this case, you can use an Index Pattern set to *revit-event*.
- List of events that passed the above filters. This is our data for the Open event.
Now that we have this data, how would we know if someone is accessing the Central File directly? Bimbeats reports a few bits of information that can tell us that.
- file.type – this field reports what kind of Revit file user was opening. This can be Central, Local, Bim360 etc. The key thing here is to make sure that the Open event was triggered for a Workshared file.
- file.path – this field tells us what’s the file path of the Central File that user was opening. If the file.type is “Central”, Central File Path will report in that field. If it’s not a Workshared file, it will just be a path to the file.
- local.file.path – since we know user was opening a Workshared file, that means that it will also have a file path for its local copy. This path is usually somewhere in your Documents. Revit makes a local copy of the Central File, assuming that you didn’t enter the Central File, in which case this local path will be the same as your file.path aka Central File Path.
Here’s what that data would look like for an event when user is in a Central File.
Same file but this time user opened it properly, and is not in the Central File.
As you can see above in the second instance my Central File Path is different than my Local File Path. That’s good.
Create a Scripted Field.
OK, so we know how to identify when users are working in a Central File. Let’s see how we can create a table listing out instances where users did that in the past. In order to do that we would first need to codify our above rule into a single boolean value, to make filtering of results easy. One way to do that in Kibana is to convert our rule into something called Scripted Field. Scripted Field is something that we can add at runtime to our data. Think about it this way. There are data entries that are saved into the ElasticSearch database. They have whatever fields/properties we set them up with. Scripted Field allows us to attach an extra property to each event, that is calculated at runtime. It’s not stored in the database though. Keep that in mind since this will be important later.
In order to create a Scripted Field we have to go to our Stack Management > Index Patterns and select the Index Pattern that we want to add the field to. Index Patterns are kind of like a filter for the data stored in the database. If we are adding a Scripted Field to the Index Pattern, it is as if we were adding that field to every data point, that actually passes the Index Pattern filter. In our case, we want to add the Scripted Field to *revit-event* Index Pattern.
- We have to give our Scripted Field a name. I am calling it “IsActualCentralFile”.
- We choose the type of value that this field will store. In our case, we want a simple Boolean – Yes/No.
- This is the actual script that will be executed for every document, to determine what is the boolean value of our scripted field.
What we are doing above is basically just checking if the fields that we need to determine our value exist in the document, and then checking if the file.path == local.file.path. That’s it. Now if we go back to our Discovery tool, and run our query, we will see an extra field available for the Open events.
Visualize your results.
Viola! Now we can easily identify any Open event where user actually entered the Central File. Let’s look at the history of that happening at Bad Monkeys. We can create a quick table via the Visualize tool in Kibana.
- Now that we have a single Boolean value that we can filter all of the events by, we can set the filter for that in our table.
- Set the Index Pattern to use when pulling data for this table. Remember that out Scripted Field was added to *revit-event* so we want to use that here as well.
- This is our Count column. We can count how many times the same file, was accessed by the same user.
- We can create a new row for each different file.path value so that we don’t count different files in the same row.
- We can also create a new row for each user/file.path so that we will get a list of users accessing Central Files, and how many times that took place for each file.
As it turns out I tend to work in Central Files quite a bit. If I am the only one accessing that file, it’s not a big deal, even if it’s a Central File. Like I said above, the issue would be with multiple people Synching to that Central File while you are accessing it. Anyways, this is a quick little table showing us the history of that event taking place, and clearly if I didn’t work from home, but instead worked at a large corporation with lots of other people, I would have earned myself the right to “Central File Access Reeducation”.
Create a Watcher.
Now imagine that you do work for a big corporation, and the idea is to prevent people from accessing Central Files directly. Just finding out about it a week later is not a feasible solution to the problem. Well, to be honest even knowing about it is a good first step, but let’s imagine we want to get notified about this issue just as it happens. No problems. Let’s look at something called a Watcher.
Watcher is a piece of functionality in Kibana that allows us to create a script that will scan the database, for incoming events, and if they pass a rule that we setup for it, then it can fire off an automatic action. It’s kind of like border security. Anyone passing through, get’s scanned, and if you are smuggling in some contraband (working in Central File), then there is a series of actions that will unfold. To create a Watcher in Kibana just go to Stack Management > Watcher.
I know, custom Watchers like this are a little heavy on code. Sorry, I didn’t say this was going to be a walk in the park. Let’s go over the script a little and break it down.
- #4 – first we need to setup a trigger. It tells our watcher how often to scan the database. Keep in mind that the more Watchers we have, firing more often, will put a burden on our database. Just FYI. In our case we want our watcher firing every 1m, so if someone opens a Central File, we should know within 1m. Nice!
- #12 – next we want to setup the Index Pattern that our Watcher will monitor. Again, we are going to be looking at *revit-event* here.
- #15-65 – this is the actual query that we want to run against the database. We don’t want our Watcher to check all of the documents, so we can use this query to define some rules around which ones to look for.
- #22 – we want it to look for “Open” events so “action.name == Open” is the rule here.
- #29 – we want to make sure that the event we are looking for has a file.type field.
- #34 – we want to make sure that the event we are looking for has a local.file.path field. All events will always have file.path field, so I am not checking for that.
- #40 – we want to make sure that “file.type == Central”. Again, Central File access issue is specific to Central Files. It won’t happen in a BIM360 file, and it won’t happen for a Local file.
- #49 – finally since we are firing this Watcher every 1m, let’s exclude events that are older than 1m.
- #55 – when all of the above passes, we can do a final check here using a piece of script, to make sure that file.path is equal to local.file.path.
- #65-72 – this makes sure that we actually found any events that matched all of the rules above.
- #73-88 – this is the action section that will determine what happens when we indeed found an event that matched our criteria. In our case we are sending off a message to Slack. We can do that in couple of different ways. I am using a Webhook to post an HTTP request to Slack. In order to do that please first add a Webhook to your Slack channel. When you do that you will know what to put on line 80.
- #89-95 – this is a transform section where I am just extracting some basic data from the actual event document that passed, so that I can then use that info in my Slack message. In this case I am extracting user.name and a file.name. When we send a message to Slack now, we can tell it exactly what user entered a Central File, and which file. It will be easier for you to take action.
Phew! This one was a handful. Sorry. Some of these things do get complicated fast. I am still learning a lot about this technology, so I am not an expert yet. Apologies if my explanations are not sufficient. I am probably still at the peak of “mount stupidity” as Adam Grant would say. It’s that place when if you started learning something like playing basketball, and you hit a few jump shots. You think you are the “shit” now, and go to the local YMCA to show off. Well, you learn really quickly that there are levels to this game, and you got long ways to go. Same here. I have been using this technology for over a year now, but it’s a lot to take in, so I am not a world renown expert just yet. Anyways, once we have that watcher setup, then we can start receiving messages in our Slack channel whenever someone enters the Central File.
Slack is not the only action that we can fire off here. Basically, anything that has an HTTP protocol, so things like Zendesk, Jira, GitHub, Teams etc. We can also send Emails or if you really want to go deep, you could probably convert an HTTP request into a Text Message. Yes, getting text messages about users working in Central File is peak “BIM Manager” mode.
Central File access is also not the only thing you can be notified about. I have examples for things like:
- user changing a Shared Parameter File Path to something that is not a company standard. It might cause an issue with Schedules and Tags.
- user unloading a Linked File for All Users. I am not sure why anyone would do that, when you can unload it for yourself, but it’s annoying when that file disappears for everyone.
- user deleting more than 10 Walls at a time. It could be anything, 10 Light Fixtures, or 100, or 1000. It doesn’t matter. The idea is to prevent some large scale deletions.
- user dragging 3D Extents of a Grid. Yeah, usually a bad idea. Some newer users might not realize that it changes across the entire model, in all views.
- user deleting a Grid. Also, not a great idea.
- user deleting a Level. Even worse idea.
- user overriding Graphical properties per View. Only annoying.
- file taking more than x-amount of time to open. It’s usually an indicator of a file that needs some love.
That list can grow and grow. There is no limit to how many you can come up with. Like I said, the only limitation might be the amount of Watchers you actually want in the database. Or you might beef up your database to make sure it’s not a big deal to run all of these Watchers on it.
More information and quite frankly easier ways to create Watchers can be found here. We covered, probably the most advanced way of creating Watchers in Elastic. There is a much easier, visual way, using their built in plugins etc. We kind of jumped into deep water here, but it also is the most flexible way to do this.
I hope this was helpful. I hope that I am not the only one that likes this technology, and that reading about it, we can adopt it a little more to our AEC needs. We are certainly trying to do that with Bimbeats.
Great as usual, article is full of useful information, was wondering if its possible to obtain sync time history from BIM360 revit files using Elastic Stack?
Yes. You can get sync time for BIM 360 files.
Great article Konrad. Can’t believe this was over a year ago. The introduction of rules and connectors within Kibana has certainly reduced the learning curve for setting up these watchers in a more friendly UI, which can be picked up pretty quickly. I created a little video on this. https://youtu.be/pOm04u-0n_A