A small business website can rank for the right searches and still lose half its potential in Google Search Console noise: pages reported as duplicates, a sitemap that points at the wrong home page, a robots.txt error on every host. None of it is dramatic, but until it is cleared, nobody can tell whether a content change helped or hurt. This post walks through one such clean-up on a Yii2 site we maintain. The fixes went out as a handful of small commits in a single afternoon.
The baseline
The property is a domain property, so it covers http and https, www and non-www, and every subdomain, including the development host. Over three months it showed a few hundred clicks from several thousand impressions, at an average position in the twenties. A few dozen pages were indexed and about half as many were not, for two reasons:
- Page with redirect. Most were the expected http to https and www to non-www redirects. The rest were retired content URLs. Nothing to fix.
- Duplicate, Google chose different canonical than user. This was the real problem.
Everything else was clean: no manual actions, no security issues, no non-HTTPS URLs. Core Web Vitals was empty, because the site has too little Chrome traffic to produce field data. For a small site that will not change, so PageSpeed Insights lab data is the only measurement available.
Trailing slashes and /index
Loading one of the duplicate URLs showed the cause. Yii2's URL manager served /about/ and /about with the same content, both returning 200, and each emitted a canonical tag pointing to itself. Google picked the version without the slash and reported the other as a duplicate whose own canonical it had overruled.
The home page had a variation of the same problem: /index served the home page with a canonical of /index, while the URL Google had indexed was /.
Yii2 has a built-in fix. The URL normaliser redirects non-canonical forms permanently:
'urlManager' => [
'enablePrettyUrl' => true,
'showScriptName' => false,
'normalizer' => [
'class' => \yii\web\UrlNormalizer::class,
'action' => \yii\web\UrlNormalizer::ACTION_REDIRECT_PERMANENT,
],
],With that in place, /about/ returns a 301 to /about. The canonical tag is now built from the trimmed path rather than the requested URL, so the two can no longer disagree. A literal request for /index is redirected to / in the controller, and the Home link in the navigation uses Yii::$app->homeUrl instead of a route that renders as /index.
The sitemap
The generated sitemap had three defects:
- It listed
/indexand not/, so it was actively submitting the duplicate. - A few indexed pages were missing.
- No entry had a
<lastmod>date. Each had a<priority>, which Google ignores.
All three were fixed in the sitemap action. The sitemap was then resubmitted, and Search Console read every page in it.
robots.txt and the development host
Search Console knew five robots.txt files (four host variants of production plus development), and each reported one error: Syntax not understood. The offending line was the same in all five:
LLMs: https://example.com/llms.txtLLMs: is not a robots.txt directive. An llms.txt file is found by convention at the site root, so the line added nothing and cost an error on every host. Removing it clears the error on all five.
The development host had been crawled regularly, yet nothing from it was indexed. Its robots.txt had no Disallow: /, and it even pointed its Sitemap line at production. What kept it out of the index was a header already set in the Apache virtual host:
Header set X-Robots-Tag "noindex, nofollow"That header was doing all the work without anyone remembering it was there. The hotfix added a dedicated robots.txt for development with Disallow: / as well, so the protection no longer rests on one line in one config file.
Two more fixes while the code was open
- Language. Every page declared
<html lang="en-US">on a site not written in English. That sends the wrong signal to Google and to screen readers. One line in the layout. - Self-serving review markup. The
LocalBusinessJSON-LD on every page included anaggregateRatingwith a hard-coded score and review count. Google's review snippet guidelines exclude self-serving ratings for local businesses and organisations, so it would never earn stars. "Valid" in Search Console only means the markup parses. The block was deleted. Stars for a local business come from its Google Business Profile.
A deployment detail that matters
The site's runtime settings live in a params.php file that the deployment tool links in from a shared directory on the server. The copy in the repository is never read in production. Any setting changed in the repository has to be applied on the server by hand at deployment, and verified afterwards. It is worth knowing which of your configuration files behave this way before you assume a merged change is live.
Why ship this separately
The development branch held a much larger release: many more pages and rewritten content. Shipping the clean-up inside that release would have changed the redirects, the sitemap and the content in the same crawl window. If rankings moved, nobody could say which change caused it. Shipping the hygiene fixes first, as their own hotfix, gives a clean baseline that the content release can be measured against.
After deployment, the remaining step is to click Validate fix on both reason rows and wait. The expected end state is a short not-indexed list made up only of host-level redirects that are meant to be there, and no robots.txt errors.
The same housekeeping on our own sites
We run the same kind of pass on pinuno.com.gh. In June 2026 we found the sitemap's cache expiry set to one second, so it was rebuilt on almost every request; it is now cached for a day. Article pages gained the JSON-LD fields Google needs for article rich results (image, url, mainEntityOfPage and the publisher logo), per-article Open Graph tags, and breadcrumbs with BreadcrumbList JSON-LD.
Yii2's URL rules shaped one of our conventions. A catch-all rule such as <controller:\w+>/<action:\w+> swallows any two-segment path, so we keep marketing pages as single hyphenated slugs at the root (for example /php-developer), each mapped to an explicit route, rather than nesting them under a bare word prefix.
And on 28 September 2026 our analytics listed "Page Not Found" as the most viewed page title for a week. Before changing any URL we read the access logs. No campaign link, published URL or internal link returned 404. The views came from mail-security link checkers fetching mangled copies of the links in an outreach email; those checkers run JavaScript, so the analytics tag on the 404 page counted them. A 404 report needs the server log before anyone changes URLs.
Two more from our other sites. In August 2026 the hand-maintained sitemap on one of them turned out to have 30 wrong URLs out of 49: language prefixes that returned 404, case-study pages that were noindex and canonicalised elsewhere, and internal /site/... route forms. It is now generated from the same URLs the pages declare as canonical and regenerated on every deploy, so the two cannot drift apart again.
And in September our energy management product showed that robots.txt is advice, not a control. Its one-click demo login was a plain link, so it authenticated on a GET, and links get followed. Bots walked it, landed as the demo user and browsed the customer portal, although robots.txt said Disallow: /. The login is now POST-only, carrying the CSRF token a crawler cannot forge, and a GET simply redirects to the page that offers the button.
Conclusion
Search Console's indexing report is only useful when everything left in it is intentional. On a Yii2 site, most of the noise comes from a URL manager that accepts more than one form of each URL, and the framework already ships the normaliser that fixes it.
If you want to learn to build PHP and Yii2 applications that search engines can read cleanly, have a look at our PHP Application Developer course.