DevXPod

DevX in Infrastructure Access with Ben Arent (Teleport)

• Chris & Pauline • Season 2 • Episode 6

Share episode

The hosts  ▻

Our guest  ▻

Things mentioned ▻

Let's chat some more! ▻

Ben:

Yeah, the developer experience for infrastructure access, I hope to some degree that there's no infrastructure access, but I think that's the best developer experience. You are always gonna need to touch infrastructure at some point for some forms, and we hope that such a smooth tool for accessing infrastructure and it provides security best practices that the secure path is the easier path to go down.

Chris:

Welcome. You're listening to the DevX podcast, the show where product and engineering leaders share their thoughts and insights on developer experience. I'm Christian Weichel joined by my co-host Pauline Narvas.

Pauline:

Welcome to the podcast, Ben. Thank you so much for joining us today. We're super excited to have you on board. How are you doing?

Ben:

I'm doing very well. Thank you. Thanks for having me!

Pauline:

Yeah. Super excited to speak to you and learn more about Teleport as well. So for those who haven't heard of you and Teleport, can you tell us a bit about yourself and what you do?

Ben:

Yeah, my name's Ben Arent and I'm a developer relations manager at Teleport. Our product itself is an open source product. 99% of it is available publicly and we have an open source edition and so we focus primarily on our community users and that can be anyone from running Teleport in their home lab to running in large organizations, which we also help keep secure.

Pauline:

I threw this question in because I wanna understand a bit more about what actually is teleport and what was the inspiration behind building it and maybe giving us like a high level overview of how it works.

Ben:

Yeah. Teleport is a access plane for accessing your infrastructure and it started off the initial idea, it was in gravity, which is our previous product, which was a Kubernetes distribution, and it was a way of accessing gravity. But before that myself and the other founders worked at Rackspace. And at Rackspace we saw this problem of sort of general infrastructure access. You'd have like hundreds of end support people accessing people's infrastructure in different pbcs, different clouds. And we saw that many of the practices for accessing that infrastructure were like outdated and had many sort of issues with them. And as teleport has evolved over time, we've worked with like many of our customers to strengthen their security. And one of the foundations of how it works is in the background, teleport is a certificate authority that is used short lived certs. And so for ssh, this upgrades your connections from public private keys to certificates. And that's the same for all of us. So for Kubernetes, we use short lived certificates. For databases and Windows desktop servers, they all use short lived credentials. And I can go into more detail why short lived credentials we see as like the best primitive for accessing infrastructure from both a security and sort of developer experience.

Chris:

So teleport issues certificates on my behalf. How does someone authenticate then with teleport?

Ben:

You join a company and you, and they have teleport your flow is pretty smooth from a developer experience. We have this command line tool called tsh and we tie TSH to your like identity provider in open source, this is GitHub. And so you log in with TSH login GitHub. You are logged in for the day and you have certificates for access for those eight hours to access whichever servers you have in your sort of R back or whichever databases. And then in the background, there's obviously like a folder which has access to these certificates, which you can use either in like open SSH or you can load it into a GUI Most people don't necessarily know this, so like when you access, let's say Postgres, you might be used to just using like a username and password, but Postgres and all databases support certificate based authentication. But it can be tricky and a bit of a pain to configure. And so most people don't necessarily go the extra steps to configure and set it up.

Chris:

So once I'm logged in, I basically get access to all my infrastructure without needing a password again. Is that right?

Ben:

Correct.

Chris:

Awesome. Why is accessing infrastructure that way? Why does that matter also, especially to developers, like, why can't I just have my one password file and pick up passwords and put them in?

Ben:

So one of the problems is as teams grow, you have lots more people joining your organization. And so let's say an old way, you join and be like, Okay, you give your public key to the ops team. They run like an Ansible playbook that gets updated. All of your servers. Okay, that might work if there's a few of you. But let's say you are growing really quickly or possibly people are churning. Do you go back and run that playbook again to remove those public private keys? And what we found is more often not people get issued access, but it's actually quite difficult to go back and remove that access over time.

Chris:

Why isn't the one like a shared password in some password managers, something not a way to go, And I think you answered that very well, is as organizations grow the requirements around who gets access to what and when grow considerably thinking also break loss scenarios, like normally you don't get access to this machine, but if you're on call and the world is on fire, Hit here, and then alarm bells go off, but you get access.

Ben:

Yeah. I think another thing we see as like a fundamental premise in teleport is identity. And so let's say if you have one password in your shared vault, you don't necessarily know who in your organization is using that password. And also if that's the one area, like humans era is basically. Main course. So this is why people do phising attacks. You pish someone, you get access to the vault. Suddenly that one password which has access to everything can be compromised. And we see the same thing with like passwords, private keys, like sessions tokens, even browser cookies can all be exfiltrated and hacker can use them to get access to systems. And so we see teleport as sort of secretless. So we don't use secrets. It's all sort of stateless and the sort of no cookies.

Chris:

Nice. Yeah, that all makes a lot of sense. Another part that makes infrastructure access difficult is usually the jumps or the hoops, or should I better say, hosts you have to jump through to get access to someplace. Does teleport help with that?

Ben:

Yeah, we see this Security can lead to security theater. And my previous large corporate job, you'd often see, like most of engineers are very smart and they'll often find ways around it. And so there might be things put in place by a security organization, but most engineers like, Oh, I could probably just develop this on like a. Let's say you put something on people's devices, they might just develop on a cloud machine, for example, to get around possible software on their own machine, or there's like other security theater that might get in the way. And so we see it by making the developer experience super smooth, it's much actually becomes easier to follow the best practices than to figure out workarounds.

Chris:

Yeah, I love this. This is a recurring theme on this podcast. Good developer experience is making it easy to do the right thing. And what you just described is exactly that. You mentioned cloud workspaces, and as at Gitpod, we're all about remote dev environments. Have you seen something like Teleport used in combination with, say, other SaaS products that need access to infrastructure?

Ben:

Yeah, we see sort of everything. I think that's the amazing thing about creating like an open source tool is like the ways in which people use your product that you don't necessarily see it. And so also the way in which people extend it as well. So people use like VS code for remote development for accessing their different remote environments. For accessing other SAS tools. One thing that we see a lot is we have application access, and so many teams build internal dashboards and tools for their own internal purposes and. Teleport can also be a way in which to protect those applications. So all of the traffic goes through Teleport. And so it makes it very easy to build like internal applications or dashboards or let's say Grafana, put it on the internet and share it with your team, making sure that you haven't opened up to the whole internet. And I think over the last couple of years we've seen a lots of compromises of like public Jenkins servers, public monitoring dashboards, public logging, even like public cloud dev environments. If there's secrets in there and you dunno who's accessing it. There's all these possibilities that there could be an area for sort of pivot and privilege escalation within someone's environment.

Chris:

How does Teleport sit in that space or in that company along things like strong VM or Tailscale?

Ben:

Yeah, so let's say we take Tailscale as a mesh vpn. They are kinda like, they connect very multiple of your hosts. The one thing that is of different with Teleports, we go very deep on the protocol and as I said, for each database, we've been slow to add our databases, but it's because we speak like the native like MySQL, Postgres. Ms. Sql, and this provides like very in depth auditing capabilities too. It's not just like that layer, it's an extra layer below for the database.

Chris:

One thing that immediately comes to mind is once you the central means of orchestrating identity and access is facilitating machine to machine communication or service accounts. How would teleport help me with that? Say I wanted to and how connect my Prometheus to Grafana the other way around. They needed to talk to each other or something to that tune.

Ben:

Yeah. We started seeing this a lot as far as like people would issue, they would use teleport and they would just, Issue a very long certificate for their like C I C D services. And the problem with issue doing like a very long certificate is you run into the same problems a long lived secret or public private keys issues around rotation become difficult. You don't necessarily has visibility into like how many of these like long lived credentials are out there. And about a year ago we developed. Sort of what we call machine id, and it's another service that issues short lived certificates for service accounts. And you can use it for both servers, clusters, databases, and it actually issues new certificates every 20 minutes and it has the full auditing. So you can see one, you have visibility into which machine, like what all machines are talking to which services. You have the ability to also lock them, if that's a possibility. And you also have the option to rotate those credentials as well. And in the backend, similar to the way in which it works, it just issues those certificates, which your applications talk natively. And databases supports certificate based authentication. Let's say if you use Ansible runs exactly the same. And for your case with the Prometheus Grafana, we recently added like tcp to native connections as well. Nice.

Chris:

Say I'm a someone on a developer experience team at a medium to larger company, chances are we're building out sort of our own ecology of tools and systems to provide a smooth onboarding and perpetual experience. What means of integrating does Teleport have to become part of such an ecology?

Ben:

As far as when people get set up. I think that an interesting thing. The year 2022 is a lot of the traditional things like SSH into machines is an anti pattern actually worked somewhere in which the DevOps guy wrote a script, like if you saw someone ssh into the machine, it was scheduled for removal after, in the next 10 minutes cuz it was seen as like a toxic host. But there are many reasons in which people in within your organization, whether it's like DevOps, dev, sec ops, but even developers might need to access a host for either pulling some logs or some debugging and the way in which we see teleport is we just wanna make that really smooth from a developer's perspective, but also make it auditing and get all of those sort of like checkpoints from security and the rest of the team. So some organizations, it actually gives them the ability to give developers more tools. And we see that for I guess it's pretty common to have like platform teams. If you have a small platform team, it's better for internal teams to build sort of tools for other developers to use and empower them to do their job. And we see teleport in the same way. You can give people actually more access with Teleport to get their job done. And on the sort of like enterprise side, we have this feature called access requests. So you can even give like zero access, but you can let people very easily request access to the infrastructure in a very smooth way. It sends a message to your like, Slack channel. It can easily be approved as well.

Chris:

Awesome. Can I have those approvals done by policy as well? Thinking of like break class mechanisms World on Fire, can't wake anyone else up. Shouldn't happen, but we all know it does.

Ben:

We have people who build like bots for different policies. And it's like an extension of our api. We actually have, like one company I know they like always order, approve it, but I think the action of developers going through the approval process makes people more, They question Should I really be getting access to this infrastructure? But they auto approve it. But you can extend our API depending upon what policy you're

Pauline:

I actually had a question that just as you were talking, it just came up. But how does I'm still trying to get my head around how teleport works with things like, for example, AWS. The I am process like how. How did those two integrate? I'm still trying to, Sorry if that's a silly question, but I don't really,

Ben:

No, it's not a silly question at all. Yeah, and I think this is kinda what we've seen. If we go back to the old days in which you'd have some servers, you'd have SSH keys, obviously now you just have AWS and you might add like key pairs for example. But AWS has a full suite of things. You may not be touching any easy to at all. You might be touching like Redshift or Lambda. And the way in which Teleport can integrate with AWS, there's a few ways. One, we support access to the AWS management console through application access. And this is of through sort of federation. And so this lets you share limited IAM roles. To your team, and let's say an example would be you wanna give your team access to CloudWatch. So you create like a very restricted CloudWatch rule. Then they can just go into teleport, they can click access to that role, and they just, it drops you straight into your CloudWatch monitoring dashboard. And so from a operational security team, you don't have to worry too much about you provide a really lockdown like cloud watch rule. And then teleport also audits what's happening. And this also works for the same roles in the command line too. So we have a TSH AWS proxy, and you can also use it for running like Terraform scripts or whatever you'd want to locally on your command line as well. And so it's a bastion for your AWS federation as well.

Pauline:

You said that teleport have an audit. Like logs. Is that similar to Cloud Trail for example?

Ben:

One fun fact that most people, I didn't know this until recently, is you should always use assume role when giving people access to AWS management console. Cuz the assume role always creates like cloud trail log. and by using Teleport it will automatically create like the assume role sort of mechanism within AWS. This is like also companies which focus just on Im, it's such a huge. Feature within AWS, which can be confusing. And I think that's where Teleport can make it quite easy cuz you can provide these like very fine grain ones. But we also have an audit log. We, our audit log is also centralized for access. It's just like adjacent form. We have a affluent DEF forwarder you can send it to whatever like SIEM tool you like. And then feed into that sort of pipeline or whatever tools customers are using.

Pauline:

That makes sense. In my old job I was a DevOps engineer and one of the main things that I had to do was take care of IAM. And I remember just banging my head against the wall every day cause it was confusing and trying to give different developers access to very specific things to the work that they were doing. Was just a bit of a headache. And now I'm like thinking of a solution like when I was listening to you talk, I was like, Oh my gosh, they should use teleport. This sounds like it will make everything a lot easier based on the scenarios that popped up during that time.

Ben:

I think also it's like best practice to create like multiple AWS accounts know,

Pauline:

obviously. Yeah. We learnt that the hard way. I've been there!

Ben:

And that whole AWS organization mapping. In the world, it's if you get your Salesforce set up incorrectly the first time, it's like a huge pain to remake it. And I think it's same with AWS. You basically have to start fresh in the right way, but people don't necessarily tell you what the right way is when you get started.

Pauline:

I've been there, I've joined a company that has used AWS as a experiment at first, and then they started migrating things over and then they didn't have the best practices and then, One of my jobs was to try and go back and implement those best practices. So going backwards and organizing AWS organizations or having separate accounts. Yeah, I've been through that pain, so I get it.

Ben:

And one thing that we see is we couldn't call 'em like access silos is just If you don't have one central source of truth for like connectivity, authorization, authentication, and audit, you're likely to drop the ball on one part of it. And so it might be good for servers, but not for your Kubernetes classes or not your databases. And that's like when the pivots and full escalation happens cuz people are like, Oh your EC2 might be really locked down, but your data science team, which is like using this like username of password, it's like poorly configured and that's how they get into your system. It can be bad if you access like a host, maybe if someone's a crypto minor, but if someone access your database with all of your customer information it's always bad news.

Pauline:

Yeah. That makes sense.

Chris:

You don't wanna make headlines this way.

Pauline:

Yep.

Chris:

How would I be using teleport in production? Like I'm using this to lock down all my infrastructure to sort of distribute access to it. Would I want to use this in production?

Ben:

Yeah, we would definitely recommend using it in production. I think it just depends who you give access to. For example, if you're using like GitOps developers likely just write their code, they create a PR, it gets deployed. It runs. No one needs access to those hosts, but there are certain team members which might need access to those hosts at certain points. And this would be an example in which you could use Teleport. So maybe you always let your security team get access, all the hosts all the time, or maybe you go through access requests. And we actually built this feature earlier this year called moderated sessions. And this works with servers and also interesting like Kubernetes, and it requires one or multiple people to observe that session. And so you can give access to your production Kubernetes cluster, but you always needed someone watching. And that person can also terminate the connection if they were to think something nefarious to haven't. So this is like the two man rule, almost like a nuclear missile silo to make sure that okay, only two people, you need two nefarious people to form the action. Like pair programming, it's good to have a second pair of eyes if you are touching production systems.

Chris:

I'm wondering how would that look like for Kubernetes? Do I see like kube control commands or how as an observer, what do I see?

Ben:

Yeah, you see a playback in your terminal of what's happening and then you can kill those sessions.

Chris:

Nice. With regards to production, like we spoke a bit about machine to machine communication, what role would Teleport play in, for example, have my server instances that scale nicely on Kubernetes, talk to say My clouds sql, or RDS instances.

Ben:

Yeah. So we just see like machine ID plan a part of just removing those like long lived secrets. And having an easy way to enroll them. And actually one way in which machine ID enrolls itself is it uses the I AWS I am identity document to enroll itself. And so you don't give it a token? It issu it. It's I identity of a machine is, comes from AWS and so there's also. Let's say it was compromised, another machine is someone else's AWS account. Couldn't talk to those RDS instances because we're proving not only identity of people, but also identity of machines.

Chris:

Sweet. Actually I want to go I almost wanna stuff to podcast, right? Go take a look at Teleport and see alright, these are the things that we could use it for. Super versatile. I'm impressed.

Ben:

Yeah. Often it's like we see this with CICD systems. The Solar Winds hack was a classic example of it was a relatively sophisticated attack under supply chain. Just shows that sometimes your building infrastructure it works, it's up and running, but it may not necessarily give it the most love and care. And I think we see machine ideas a way of just. Strengthening the security of those communications between your machine, the C I C D machines and your production hosts. And also just making sure that the identity of those hosts are the ones pushing production code to which other systems or however they're communicating, which you often hear zero Trust is mostly around sort of users. We assume the same zero trust between like machines to machines is making sure that you trust the machine. Talking to your database is the right machine and not just another machine that has the password.

Pauline:

More broadly speaking, I wanna take a step back and focus a little bit on developer experience and where you see the space evolving. So what would that look like in two to five years?

Ben:

Yeah, the developer experience for infrastructure access, I hope to some degree that there's no infrastructure access, but I think that's the best developer experience. This is also like the Heroku esque you just write your code, you ship it, you deploy it. But you are always gonna need to touch infrastructure at some point for some forms, and we hope that developer is such a smooth tool for accessing infrastructure and it provides security best practices that the secure path is the easier path to go down. instead of going through a less secure like password based authentication. And this is why we're seeing people using teleport in their home labs is that they like, it's just such an easy way for them to add like a multiple rise pies to the network and get access to them. It is just actually quicker to use teleport than using other means of setting up access to their infrastructure. And that's where we hope to be in the next five to ten years, is just when you do access your infrastructure, teleport is the quickest and easiest way to.

Pauline:

So essentially developers don't really need to think about it because it just happens and it's like you said, the easiest path to follow. This episode has gone by so quickly. Thank you so much for being such a fantastic guest. So to end this podcast, we usually ask our guests about one thing they'd like to shout out about recently. So something they're either learning or someone they want to shout out that has impacted them. It could be tech or non-tech related use of space. It's all yours. What would you like to shout out about this week?

Ben:

We do have our own community event coming up. This is a, I guess like shameless plug for Teleport itself, but we have our first user conference, which is gonna be in San Francisco on November 9th. And I've been rallying a whole bunch of speakers and I think this kind of goes back. Interesting ways in which Teleport is being deployed. We have a couple of self-driving truck companies who've deployed Teleport so they can access it on the road. We have people using it for their data scientists. We have people in like high performance computing. To some degree like consideration upon like some cloud providers. But we're actually seeing like more compute go to the edge. And this is like another way in which Teleport has helped people. And I think it was really excited about just computing, growing and like the edge possibilities. And so hopefully we'll have a lot of these stories for Teleport Connect, which be November 9th in San Francisco.

Pauline:

That is so cool. You've just given me an idea for a community event we should have . That's fantastic. I hope it goes really well. Thank you for sharing that. We'll also share the links in the show notes for anyone who's interested in attending or just checking it out. Chris, what is your recommendation this week?

Chris:

So I've been going back and forth between a book or the build system that Gitpod uses, but I think I'll go for that. So Gitpod uses a build system called Leeway. We'll put a link in the show notes and it's heavily inspired by things like Basil and all the million clones that it spawned. And it's yet another one except that it doesn't try and pull you into this world entirely. But it's super opinionated. It works for Gitpod. I'm not sure it's gonna work for anyone else, but the thing it does really well is it gives you a lot of similar benefits that things like basil give you so cing and. Reasonably controlled builds and almost reproducible builds in our case. But you can still use your yawn and your go and whatever tool you have, like you normally would, and so you don't get fully pulled into sort of this world of the build system. So shout out to Leeway.

Pauline:

That's awesome. For me this week, it's actually KubeCon . Because I know this episode will be out before we all go to KubeCon but KubeCon na is happening in Detroit and Gitpod have a booth there. So feel free to come and chat to us. We've got a team prepared and ready to give you some cool demos and to chat to you about how you can use Gitpod for your teams, for your own projects. And yeah, come and check us out. We're always happy to have a chat.

Chris:

Can I add to that? We also have two talks this year. Gitpod has two talks this year at KubeCon. One where we talk about making image pulls faster using ipfs. And another, which I'm really excited about called Kubernetes, where we talk about how bees have solved problems of distributor systems millennia ago. So come and check us out at KubeCon!.

Ben:

We have a talk as well. So for the Cloud Native Security Conference, which I guess is like the day before on machine, ID. And so we're talking about giving machines identity and accessing infrastructure securely. So we'll be a supert deep technical dive into machine id.

Pauline:

Might go and see Ben if you're there. Yeah, I'll see you there. Yes. Amazing. Yeah, it's the co-located events the day before the actual KubeCon conference. So if people want to get started with Teleport and find out more where can they go?

Ben:

You can just go to GitHub, Search Teleport. As I said, it's an open source, open core product. All of our issues, all of our developments, even our design documents are all in the public and open. We believe that's very important for a security focused product. And if you want the more kind of corporate spiel, you can go to go teleport.com and we have the download there that you can just download it, try it on a raspberry, try it on a host. Try it on a Kubernetes cluster.

Pauline:

Awesome. Thank you so much, Ben for joining us on this episode. I'm really excited for everyone else to hear this.

Ben:

Okay. Thanks for having me.

Chris:

Yeah, thank you very much.

Pauline:

Thank you for listening to this episode of DevX pod. Want to continue the conversation about developer experience? Head over to our community Discord at gitpod.io/chat. We have a dedicated channel there for us to talk all about DevEx.

Chris:

To make sure you don't miss the next episode followers on your favorite podcast platform or subscribe to our newsletter DevEx Digest. You can also find out more about Gitpod on Gitpod.io. Start a new workspace and tell us about your developer experience. See you in the next episode.

People on this episode

Head Shot

Christian Weichel

Host
Head Shot

Pauline Narvas

Co-host