University
Home
Search
⌘E
Dashboard
Try Webflow — it's free
Learn
Courses
Docs
Interactive Learning
Glossary
Certifications
Engage
Support
Community
Events
Forum
Accessibility
Switch to Light mode
Switch to Dark mode
High contrast
Glossary
Robots.txt

Robots.txt

A robots.txt file instructs search engine crawlers which webpages and sections they can or can't crawl and index. They exist in a website's root directory (the topmost folder in a file system) and regulate how search engine crawlers should behave while indexing a website.

Think of a robots.txt file as traffic signs for crawlers — steering them toward authorized areas while cautioning against others. Web developers create robots.txt files to regulate how search engines interact with websites, allowing site owners to protect sensitive information, prevent content duplication, and keep maintenance areas hidden.

However, robots.txt files have specific limitations. For example, not all search engines support robot.txt files — Google’s crawlers are likely to follow a file’s rules, but other crawlers may interpret syntax differently. As a result, understanding index-blocking methods, such as the “noindex” directive and the correct syntax, is vital for protecting website information.

Visit Webflow University to learn how to create a robots.txt file for your website.

Relevant courses

CMS & dynamic content

Learn how to use Webflow CMS to build completely custom websites powered by dynamic content, and define your content's structure and style.

Figma to Webflow

Learn the entire design process from idea to final output as we take you through Figma, Cinema 4D and Octane, and Webflow.

Interactions & animations course

Learn how to animate multiple elements managed on a single timeline — bring your designs to life with rich, sequenced interactions and animations.

CSS styling

Take a closer look at all the creative styling possibilities that CSS offers. Learn how to style elements visually, with the power of Webflow.

CSS grid in Webflow

An overview of CSS grid. Use the grid layout in Webflow to get more control over your designs and create powerful, responsive layouts — faster.

Ultimate web design course

From the fundamentals to advanced topics — learn how to build sites in Webflow and become the designer you always wanted to be.

Other glossary terms

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Gutters

→
Gutters refer to the whitespace between content columns on a webpage.

Resolution

→
Resolution is a metric measuring clarity and detail through the number of pixels displayed on a screen or used to render an image.

Constant

→
In coding, a constant is a fixed value that can’t change, no matter where it appears or what the program uses it for.

Content management system (CMS)

→
A content management system (CMS) is a tool for creating and managing a website and its content.

Decryption

→
Decryption is the process of reverting encrypted data to its original format.

Adobe Fonts API token

→
An Adobe Fonts API token protects and supports your data from Adobe Fonts with other programs you may be using to personalize your website’s fonts.

Cipher

→
In cryptography, a cipher is a digital security algorithm that encrypts and decrypts data.

CRLF

→
On Windows-based systems, the sequence of two ASCII characters representing a line break in text files is called "Carriage Return, Line Feed," or CRLF.

Primitive

→
A primitive is the simplest JavaScript data type, representing fundamental values that do not encapsulate any object, method, or property.

Strong text

→
The <strong> element defines important text with added semantic importance.

Domain name forwarding

→
Domain name forwarding is a process that automatically redirects visitors from your domain name to a different web address.

Key

→
A key, or web key, is a unique identifier that authorizes access to a webpage or service.

Mutable

→
In JavaScript, mutable objects are dynamic elements that allow post-creation changes and modifications.

Flex-align

→
A flexbox property that lets you customize the alignment behavior of flex children, based on the cross-axis of the flex container. The cross axis is the opposite of the flex-direction property, so if you set the direction to vertical, the cross axis is horizontal. Options include start, center, end, baseline, and stretch.

Cipher suite

→
A cipher suite is a collection of encryption algorithms that work together to secure a network connection.

Right margin

→
Right margin is a CSS property that defines the space between the right edge of an element and its surroundings.

Made in Webflow

→
Made in Webflow showcases Webflow sites made by individuals and teams.

URL

→
A Uniform Resource Locator (URL) is a standardized web address format that allows users to identify and access an online resource.

Front-end

→
The front-end refers to the visual and interactive part of a website or application that users directly interact with.

Height: auto

→
The "height: auto" CSS property automatically adjusts an element’s height based on its content.

Paginate items

→
This checkbox allows you to enable or disable pagination on the selected Collection list.

Static content

→
Static content is content that’s not bound to any Collections or dynamic elements.

Return on investment (ROI)

→
Return on investment (ROI) is a metric evaluating the profitability or gain of a venture.

Margin

→
Margins are an essential spacing element of web design, dictating the amount of whitespace between a box and its surroundings.

Clickjacking

→
Clickjacking is when a malicious actor deceives website visitors into clicking buttons or links hidden in a fake layer to steal sensitive data or install malware.

Static page

→
A static page is a document with fixed content that makes up an individual web page on your site.

Child

→
A child element is an element nested within a parent element in a hierarchy.

Top margin

→
Top margin is a CSS property that defines the space between the top edge of an element and its surroundings, such as other elements, the browser window, or the edge of a container.

Automagically

→
Automagically describes complex content or functions that appear as if they were created using magic.

Tabs

→
A Webflow component that displays a pane of content associated with a tab menu button.

Undo

→
To move backward one step while making changes to a Webflow site.

Basic section

→
A section of the Add panel that contains the elements that act as a website's fundamental building blocks (e.g., div block, list, button, etc.).

Page name

→
The title of a page in your site. This is visible in the Pages panel.

Dashboard

→
From your Dashboard, you can access, create, and manage all your sites.

Pagination settings

→
Through the pagination settings, you can preview Collection list pages by choosing the page number from the page dropdown list, or you can switch to the previous or next page using the arrow keys.

Certified

→
A digital certificate is a data file embedded in the site that signals the site’s safety to visitors, internet service providers (ISPs), and servers.

Display: block

→
Display properties tell browsers how an element should behave on the page. The most common display property keyword category is display: block.

Backlink

→
A backlink, also referred to as an inbound link, is a hyperlink from one website to another website.

Descriptor

→
A CSS descriptor sets specific properties and values of an at-rule (@-rule) element, dictating its appearance and behavior on a web page.

URI

→
A Uniform Resource Identifier (URI) is a string of characters identifying an online resource.

SSL CDN

→
SSL CDN, or Secure Sockets Layer Content Delivery Network, is an option that allows you to serve your images over your own CDN service while using SSL.

Flex-direction

→
Flex-direction is a CSS property that defines the main axis and direction of flex items, dictating the arrangement of the items within a flex container.

Localization

→
Localization (l10n) involves adapting digital content and applications to meet a specific region or culture’s language and conventions.

Password protection

→
Password protection is a security feature restricting certain website areas to authorized users, requiring a secret phrase or key to access them.

Semantics

→
Semantics accurately represent the structure and purpose of content in computer languages, enabling web developers and programmers to ensure the written code produces an output legible to humans and machines.

Selector

→
A selector identifies and selects specific HTML elements to apply CSS styles to.

Minimized CSS

→
A method of lowering the file size of your stylesheet(s) by removing all white space from your CSS files.

Inline style

→
Inline style applies CSS to an individual HTML element using the "style" attribute within the HTML tag.

Height

→
Defines how tall an element can be. This can be defined in absolute terms (pixels) or relative terms (ems, rems, percentages, viewport-height, or viewport-width).

Combo class

→
Combining two or more classes into a single selector to target an element.

Inline Frame (iframe)

→
An inline frame is an HTML element that embeds an HTML document within another webpage.

Username

→
The ID used to log into your Webflow account.

Abstraction

→
In object-oriented programming (OOP), abstraction conceals irrelevant details so they don’t interfere with the task at hand.

.htaccess

→
.htaccess is a powerful configuration file that allows you to configure and control how your website functions.

Utility pages

→
Utility pages are default, customizable templates for your site’s 404 page, Password page, and Search results page.

Image field

→
Available in CMS Collections. With this field, you can upload images from your computer and use them as dynamic items in your individual Collection item.

JSON

→
JSON, short for JavaScript Object Notation, is a human-readable, lightweight format for recording, storing, and exchanging data.

Forms section

→
A section of the Add panel that houses form elements like inputs, checkboxes, and radio buttons.

Dynamic content

→
Content you create once, then publish in many different places across your website. As you update this content in the CMS, it automatically updates wherever it exists on your site, making it convenient to create and manage larger websites.

Sidebar

→
A sidebar, typically positioned on the left or right side of a website or app, is a vertical component that contains supplemental information to enrich a webpage’s primary content.

Search Engine Optimization (SEO)

→
Search Engine Optimization (SEO) is a collection of strategies to enhance a website’s ranking and visibility on search engines.

Display: none

→
Display: none is a CSS property that hides website elements. This code tells the document flow to ignore the element entirely, making it invisible.

Mobile (portrait) breakpoint

→
A Webflow canvas setting that lets you create styles for viewports with a minimum width of 240px and a maximum width of 479px. Also known as the mobile portrait media query.

Synchronous

→
Synchronous programming refers to sequentially executed operations.

API

→
APIs, or Application Programming Interfaces, are common code protocols that let apps safely exchange data with other apps, software, and hardware.

Sibling

→
An element that shares the same parent of another element.

Middleware

→
Middleware is software that links different programs and applications, allowing them to interact and communicate effectively.

Navbar

→
A navbar, or navigation bar, is an interactive user element with a menu or a set of links to a website’s main sections.

Traffic

→
Traffic, or web traffic, refers to the number of users who visit a website over a period of time.

Gamut

→
Color gamut describes the full range of colors a device can reproduce on its display.

Conditional

→
Conditionals, also known as conditional statements or expressions, are coding instructions that tell a program to execute an action for a certain condition.

CRUD

→
CRUD stands for Create, Read, Update, and Delete — the four basic operations all software applications should perform.

Left margin

→
Left margin is a CSS property that defines the space between an element's left edge and its surroundings.

Design mode

→
The mode that enables a designer to build in Webflow.

Responsive design

→
Responsive web design (RWD) is a web development and design approach to create websites, applications, and user interfaces (UIs) that display and function effectively on any device or screen size.

IP address

→
An IP (Internet Protocol) address is a unique string of characters identifying devices connected to the internet.

Email template

→
Email templates — or pre-designed HTML layouts — ensure consistent branding and visual aesthetic in marketing campaigns.

Link block

→
A link block operates much like a div block, an HTML element for grouping and organizing other HTML elements (like text, images, and forms) into block-level containers. But in the case of a link block, what’s inside the block becomes a link.

Web Accessibility Initiative (WAI)

→
The Web Accessibility Initiative (WAI) is a World Wide Web Consortium (W3C) project to enhance accessibility for users with disabilities.

CSS preprocessor

→
A CSS preprocessor is a program that helps developers write Cascading Style Sheets (CSS) by providing advanced features such as variables, functions, and nesting.

Switch field

→
Available in CMS Collections. This field offers you a powerful filtering tool for Collection items. For example, you could create a switch labeled "Featured?" for Blog Posts, and then switch on the toggle to showcase particular posts.

Subdomain

→
The part of a domain that comes before the root domain (e.g., blog.yourwebsite.com is a subdomain whereas yourwebsite.com is the root domain).

Meta title

→
A meta title is an HTML element that defines a web page's title and appears in browser tabs and search snippets on search engine results pages (SERPs) as a clickable headline.

Collection list layout

→
Lets you control how Collection items display inside a Collection list. The default layout option has each item covering the full width of the parent element. You can change this to display Collection items in 2, 3, 4, or 6 columns.

Component Module

→
A section in the Add Panel that houses pre-built Webflow elements such as sliders, tabs, and lightboxes.

Webflow badge

→
A visible "Made in Webflow" badge displayed in the bottom right corner of Webflow sites on Starter plans.

Webflow forum

→
The place to go when you need help with your site's design, custom code, etc. Visit the forum.

Cryptanalysis

→
Cryptanalysis examines cryptographic systems to find weaknesses that could allow attackers to uncover plaintext or break encryption.

Encapsulation

→
Encapsulation is an object-oriented programming (OOP) process that combines functions and the data they manipulate into a single entity.

Total asset size

→
The total weight of your site's HTML, CSS, JS, and images.

File Transfer Protocol (FTP)

→
File Transfer Protocol (FTP) allows for the transfer of files between two parties over TCP/IP-based communication channels, such as the internet.

Alt text

→
Alt text, short for alternative text, refers to text that describes online images.

Bounding box

→
A bounding box is a rectangular border around a website element (e.g., a heading, paragraph, or image) that appears during the design process.

Headless CMS

→
A headless content management system (CMS) separates the front-end presentation layer from the back-end management layer, allowing for more flexible content delivery.

Call-to-action (CTA)

→
A call-to-action (CTA) is a website feature that prompts visitors to perform a specific action.

Call stack

→
A call stack is an ordered list of the functions (or stack frames) a program has to execute before it returns a result.

Webflow CMS

→
Webflow's content management system (CMS) is where dynamic content (Collections and their Collection items) is stored and maintained so you can reference it throughout your site. A CMS lets you input data in one location, then use Collection template pages or Collection lists to display that content without having to update the content displays one at a time.

BigInt

→
BigInt, which stands for “big integer,” is a JavaScript data type. JavaScript uses data types to decide how to treat a piece of data.

SCA and PSD2 Regulations

→
Mid-September 2019, new European requirements for authenticating online payments were introduced in Europe as part of the second Payment Services Directive (PSD2). The Strong Customer Authentication (SCA) requirements apply to businesses in the European Economic Area (EEA) that accept online card payments from cardholders whose banks are also located in the EEA.Webflow Ecommerce payments are PSD2 and SCA compliant (i.e., support 3D Secure payments) for European customers.If you’re a merchant in the European Economic Area (EEA), all you need to do to ensure that your Webflow Ecommerce store is PSD2 and SCA compliant for your European customers is to republish your store if it wasn’t published on or after September 11, 2019. Read more about how to make your ecommerce store compliant in our Payment gateway article.

Certificate authority

→
A certificate authority (CA), also known as a certification authority, issues digital certificates that verify a website’s owner.
Hmm…we couldn’t find any results. Try a different search term or reset the filter.
Reset the filter
Load more