I have been spending more and more time on Hybrid SharePoint recently. I’ve been working with hybrid for over 3 years, but it feels like we’ve now reached a tipping point.
Last week’s SPTechCon included many talks featuring hybrid, which inspired my blog article “Hybrid Everywhere”. Last week also was the TAP kickoff for the upcoming new Hybrid Search Crawl Appliance (you’ll be able to learn about this in detail at the Ignite Conference in this session: Overview and Implementing Hybrid Search with the New Hybrid Search Crawl Appliance); the kickoff felt like old home week because of all the friends, former colleagues and customers that were there. Participation was very high at sessions I’ve done on Hybrid SharePoint recently, and the rate of requests I am getting for consultation on hybrid strategy is going up fast. There’s a lot going on with Hybrid SharePoint, and a lot of interest.
Usually when I do a session around Hybrid SharePoint I focus on the scenarios, user requirements, and organizational considerations first, before getting into the architecture, strategy, and bits and bytes. For this article, though, I’m going to use the “GBU” lens. You may know that this term comes from an epic 1966 Italian Spaghetti Western file Il buono, il brutto, il cattivo, literally “The Good, the Ugly, the Bad”) – starring Clint Eastwood in his early career when he was a huge name in Italy but not yet in the US. But when I worked at Microsoft covering “the GBU” was an essential practice and the habit stuck with me.
The Good: Enjoying the Best of Both Worlds
Combining SharePoint Online (in Office 365) with SharePoint Server (on premises, in Azure, or in a hosted offering) is an important strategy for organizations seeking to move to the cloud and/or leverage the latest Microsoft offerings. The promise of Hybrid SharePoint is that you can enjoy the “best of both worlds” benefits of Cloud adoption sooner, and have lower risk and increased flexibility. Though there is plenty of hype around this promise, there are genuine benefits to be gained.
Real Cloud Benefits at Lower Risk
The benefits of using SaaS for productivity software are real, and IDC reports that 70% of CIOs will embrace a “cloud first” strategy in 2016. Those that have adopted SharePoint Online have reported increased efficiency, improved employee mobility, and increased ability to innovate as an organization. IT groups report that their staff is freed up for other projects, their operating costs are reduced, and they are better able to offer new products and services.
Most large organizations have a big investment in on-premises systems, including big SharePoint farms with customized business-critical applications. Moving everything to the cloud overnight is just not feasible, and even if was possible it would be an incredibly risky proposition. Hybrid SharePoint reduces this risk. You can maintain existing on-premises systems and customizations, and migrate one step at a time. You can also meet regulatory, performance, or functional requirements that require on-premises SharePoint and aren’t covered by Office 365 today (more on this later).
Many people I talk with also cite the ability to keep up with Microsoft’s “cloud-first/cloud-only” approach as a big benefit and as a risk reducer. Microsoft’s strategy has meant a dramatic change in the way new capabilities are announced and released; nearly all customers and partners are struggling to adapt and a hybrid strategy makes this much easier.
Hybrid Increases Flexibility
As I discuss in my webinars and seminars, many people think of adopting hybrid in the context of migration – it is a stepping stone to a future where everything is in the cloud. But the co-existence strategy can be very effective. You might consider maintaining a hybrid model indefinitely; that way you can continue using on-premises systems and customizations as much as you like, and mix them with O365 according to your needs and your pace.
The co-existence approach also provides flexibility. I know several organizations using a hybrid strategy as a way of keeping their eggs in multiple baskets. Having all of your data in a single place makes you vulnerable, and dependence on a cloud vendor goes deeper than dependence on a vendor of on-premises software. Fredrik Schmidt of accellerite maintains that “enterprises need to date clouds, not marry them”. I love his blog article about how cloud vendors are the crappy spouse in an unhappy marriage. Once you move everything to any vendor’s cloud, it is very hard to move off, so keeping a toehold on-premises gives you flexibility.
Ongoing Big Investment in Hybrid by Microsoft
Microsoft has made a major investment in hybrid. There has been consistent and regular improvement and Microsoft is working hard to make the building blocks for Hybrid SharePoint solid. There were some major gaps when SharePoint 2013 first came out which made it nearly impossible for people to field successful ‘split-workload’ configurations (where the same user worked on sites in SharePoint Online and also on sites in SharePoint Server). SP1 was a big improvement; it made deploying federated identity practical and included many other fixes for hybrid configurations.
You should expect a major investment in hybrid from Microsoft with SharePoint 2016. Julia White, who is responsible for the SharePoint business at Microsoft, announced this as part of her “Evolution of SharePoint” article on the Office blog two weeks ago. In some SharePoint’s evolution follows the path that Exchange has taken; hybrid Exchange configurations started with Exchange 2010 and were initially quite painful but are relatively routine now. In other ways SharePoint is now leading the way as cloud-only capabilities like OneDrive, Delve, and PowerBI grow and the need to tap them in a hybrid environment becomes evident.
No matter how you look at it, Microsoft’s large investment is good news for hybrid. It is a key differentiator for them, so you can feel confident that it won’t be dropped and that it will be well-supported.
Smooth Sailing for Some Scenarios
Once the basic elements, especially Federated Identity, are established, there are elements of hybrid SharePoint that just work. Many of these are of the “split workload” variety. You can mix SharePoint on-premises with Exchange Online and Lync Online, and features such as site mailboxes, presence, and contact cards light up with no trouble. Yammer can now be integrated smoothly with SharePoint 2013 on-premises; if you haven’t done this already check out the technet guidance and documentation, which came out a couple of weeks ago.
Another example of smooth hybrid I like is the ability to expose content to external users. SharePoint online makes it much simpler to provision extranets and manage access to corporate content for remote employees, partners, suppliers, and customers. I recommend Sharon Richardson’s blog on SharePoint 2013 and external users if you want to get familiar with this. Extranets also function in a hybrid environment, so you don’t need to copy content across farms or worry about where it lives when you are collaborating with people outside your company.
BI also works pretty well in a hybrid environment, at least for mainstream use cases. I’ve been working with the PowerBI previews for a year. I love it, and use it regularly in production. This August, Joe D’Antoni and Stacia Misner published a helpful article on using PowerBI in a hybrid environment. You can read about and download it from the “User Ed” MSDN blog.
Strong Training and Technical Material Helps You Ramp Up Quickly
Another positive is the availability of quality training and documentation to help with implementing hybrid. Some great documentation, and guidance came out over the summer from Microsoft, which you can reach through the Hybrid SharePoint Center on Technet. MSDN published a nice series of SharePoint Hybrid Training Videos this fall, and there are classes on Microsoft Virtual Academy as well as through many training companies and conferences.
It is also possible to get hands-on quickly with hybrid SharePoint. A free trial of Office 365 is very simple to provision; I recommend this as a way to learn and practice before you change your production tenant. Technet has a hands-on lab published, although hybrid hasn’t made it to technet’s Virtual Labs
Directions on Microsoft has also come out with some fabulous analysis and recommendations (read DoM’s analysis of Hybrid SharePoint), and I recommend their Office 365 Evaluation Guide as a concise, clear, impartial approach that helps a lot when you are looking at Office 365 adoption.
The Bad: Added Complexity & Confusion
There’s a lot of goodness to Hybrid SharePoint, but all is not sweetness and light. I think of this as analogous to cross-breeding in farming (I grew up on a farm, and still raise hybrid strawberries). Although heterosis, aka ‘hybrid vigor’, is very common and provides some remarkable strengths, it’s also common for crossbreeds to also have the weaknesses of the breeds from which they descend and for there to be a few extra issues too. So there’s definitely a Bad and Ugly side to Hybrid SharePoint.
More Moving Parts Means More Work
It’s no secret that a hybrid SharePoint/Office365 environment poses very real challenges. It can be very time- and labor-intensive to set up. It’s complex to manage because there are two separate but equal segments requiring support. Anything hybrid is by definition more complex than something that is not. You are tying together multiple systems that are each complicated in their own right.
Bill Baer, one of my heroes and the guy behind many great things in SharePoint, is the technical product manager at Microsoft behind hybrid SharePoint. Last week he gave out a preview of the SharePoint Cloud Migration Assistant to a set of partners. This is a cool tool with a wizard that walks you through a customer’s scenario, asks lots of questions, and lays out highly specialized and in-depth plan to get started with a SharePoint hybrid deployment. I really like this tool and it’s an example of one of my “good” points – Microsoft is serious about making Hybrid successful. But the mere fact that you need a deep tool and detailed workshop underlines the complexity involved. The tool encapsulates many technical and licensing prerequisites and factors that require design changes and mitigations…some that I didn’t know about before even though I live this stuff. So – hybrid is more complex, and there are a number of gotchas and surprises to watch out for.
In November, Microsoft published a list of the top operations and management issues for SharePoint Server 2013 hybrid. This gives you a sense of what can go wrong and a flavor of the extra work involved in making Hybrid work.
DirSync and Proxies and Certs, Oh My!
One of the cornerstones of hybrid with Office 365 is identity federation. This is how both Office 365 and your on-premises SharePoint farm know who you are to give you a secure, unified experience. You need to configure your hybrid environment for single sign-on and server-to-server trust. Then you set up directory synchronization and identity federation. Depending on the authentication topology, you may also need to set up a reverse proxy (so that users online can access information that resides in SharePoint on-premises).
Talk about a lot of moving parts! This process needs SSL certificates, Azure AD setup, and machines for the reverse proxy and the directory synchronization. You have to configure the on-premises security token service, understand how ADFS and OAUTH work, and keep everything up to date with patches and versions. If admin passwords change, you have to rerun synchronization. You need two different valid accounts for DirSync to work, and you need to be sure every user has a valid UPN (user principle name) and valid licenses. Then you need to remember to administer federated users in the on-premises AD, but to administer external users in Azure AD. Just thinking about this whole process makes me tense, it is anything but easy.
When SharePoint 2013 first shipped, this process was so broken that the vast majority of people failed to get things successfully set up. It was indisputably UGLY (it didn’t help that the only reverse proxy supported at the time was Microsoft’s TMG which was already past end-of-life!). Luckily that is now a distant memory (or nightmare). Those issues mostly got resolved and it really is possible to walk through a configuration with DirSync, a reverse proxy, and the security token service (STS) and come out with a working system.
However, this process is still complex, time-consuming, and error-prone, and clearly belongs in the BAD bucket. Just as in the Wizard of Oz, you have to follow the yellow brick road from end to end; I recommend repeating “DirSync and Proxies and Certs, Oh My!” until you are out of the woods and reach the Emerald City.
That Sync-ing Feeling – No Sharing Service Applications with Hybrid
SharePoint 2013 allows for some service applications to be shared across farms, with the concept of a publishing farm and a consuming farm for. This is a very nice facility and it’s available for the services I consider foundational – User Profile, Managed Metadata, Secure Store, Machine Translation, Business Data Connectivity, and Search. You can’t do this across a hybrid configuration, and the coverage across these service applications varies:
• Machine Translation is a service simply not supported in Office 365
• Business Data Connectivity and Search are have some new facilities explicitly meant for Hybrid configurations; both of them have some significant warts ranging somewhere between BAD and UGLY.
• Managed Metadata isn’t shared at all between on-premises and Online, which is UGLY (more on this below)
User Profile data often goes well beyond what is covered in DirSync, for example with My Sites. Users end up with two user profiles, which is confusing. In many situations there are cloud-only users, but there are also a variety of customizations people do on-premises related to user profiles. You can end up with custom code to hide on-premises profiles and synchronize profiles across the hybrid configuration. Management of this situation can be complex and is clearly BAD.
Finding Issues with the OOTB Hybrid Search Experience
The OOTB hybrid search experience today is lacking. It is built on remote result sources and a result block construct as shown below. However, this turns out to be insufficient – it just isn’t what users expect. There are three main drawbacks, written in red below. I have met many people who fall back to having two separate tabs for search, and users overwhelmingly tell me they aren’t happy with this experience. As a search nerd, this pains me.
OOTB Hybrid Search Experience
I would be tempted to put this in the UGLY category, but will keep it as BAD because there is hope. There are products from partners (including my company, BA Insight) to address this and provide a great hybrid search experience. There is also a new hybrid crawling appliance in the works that will alleviate some of this problem (though it will also need complimentary partner products in many cases).
This is my favorite subject, but I’ll keep it brief here. Feel free to reach out and let me know what you think around the hybrid search experience.
Migration and Configuration Management
Migrating to a hybrid configuration and managing the configuration once you are running can be complex tasks – because of the “many moving pieces” problem described earlier. In this area as well there are gaps with the OOTB hybrid solution. Microsoft doesn’t provide any kind of SharePoint migration tool, but does offer two manual SharePoint migration tips. Although Microsoft has made significant progress in its migration processes, the migration process for going from SharePoint 2013 to Office 365 still remains overwhelming.
There are also a variety of feature and configuration differences that trip people up and result in a lot of confusion and fatigue for those on the journey of Hybrid. Two examples that I see people surprised by are lack of email-enabled lists and libraries, and lack of claims-based authentication in SharePoint Online. Some features won’t be missed, depending on your plan and needs. But if you work with SharePoint and have a good amount of data held within it, you might want to consider a third-party solution if you cannot migrate the data manually and/or if you have a complex configuration.
The Ugly: OOTB is Just Not Enough
When talking about something you know and love, it’s hard to describe the truly ugly parts. I am a Prius driver and often reflect on my experience with hybrid cars and the unexpected benefits as well as the ugly parts. I love my car and really admire the success the model has had – but I do still recognize that it has a stupid name, ugly styling, an interior designed by aliens, and lousy performance. Oh, and it is full of contradictions when it comes to environmental impact and has spawned more than its fair share of green-ist righteousness.
There are some pretty ugly aspects of Hybrid SharePoint, too. The good news with software, though, is that there are always solutions. For the issues in the UGLY bucket the solution is nearly always something added on – a third party product or custom code.
Managed Metadata – Managed Where?
If you have content in two places you really want to be applying consistent metadata. Metadata drives findability, workflows, and many governance and compliance activities. But there is no OOTB facility to synchronize term sets between on-premises and Online.
In part this is because it’s hard to do. The Managed Metadata Service is built on GUIDs and is much more slippery to synchronize and manage than User Profiles. It is also a bit of a specialty – not everyone uses managed metadata – and perhaps a bit lower in Microsoft’s priorities as a result.
A couple of us vendors and consultants went ahead and created tools to take care of this issue, with Microsoft’s encouragement. I’d recommend using them, and considering one of the tools that create metadata by machine as well. You will also find that you want to have some ownership and simple policies around metadata management – it should be somebody’s job.
Integrating External Data into SharePoint Online
If you look through the documentation, it looks as though you can bring external data into SharePoint rather easily through BCS. And in fact, you can surface external lists, albeit with a fair amount of pain. With BCS installed on-premises, connectivity to the external data source from the on-premises instance, and two-way authentication and external URLs configured, you can make this work. And there is a cloud-only authorization and data flow that gives access to content in SQL Azure.
So why would I consider this item as UGLY when something like the OOTB hybrid search experience is BAD? OK, perhaps I am being a bit arbitrary and should reverse them, but here are three gotchas:
1. You can only use OData data sources. Not a big limitation but it means redoing the BCS models you may have previously built.
2. Performance is slow, and the configuration is a bit fragile. I have run into several people with enough complaints about this that they gave up.
3. You can’t surface this data via search. If you’re familiar with BCS you already know that a BCS model lets you show external content as a list item but doesn’t automatically give you search over the external data. But unlike the on-premises implementation, there is literally no way to index content from outside Office 365 (actually, from outside SharePoint Online).
There is also a pretty cloudy feeling roadmap in this area (pun intended). The BCS-based external list model is different with on-premises data versus SQL Azure. The Hybrid Crawl Appliance addressing search of external content uses a different flavor of BCS. PowerBI uses a different mechanism, the Data Management Gateway. And the Azure Data Factory has yet a different way to bring in external content to the cloud. How this will all pan out will be very interesting but it is not for the faint of heart, so I would recommend getting some help and/or buying something that makes it your vendor’s problem rather than yours.
Reporting and Analytics
Capturing usage and adoption metrics from Sharepoint Reports
Hybrid deployments add a new set of issues around capturing usage and adoption metrics. Some of these are intrinsic to looking across a more complex system, some are based in the different physical locations of the data needed to support analytics. Other issues are due to analytics that are weak to begin with.
My colleague Joel Olsen wrote an article on the changes in the out of the box analytics for SharePoint 2013 called “SharePoint 2013 Analytics A Big Step Backward.” In it, he described how SharePoint Search works to generate SharePoint analytics and lamented the gutting of the analytics components from the platform. His message was clear — capturing SharePoint analytics is best left to third party vendors that specialize in this area.
SharePoint Online analytics are even weaker than those on-premises. In a hybrid environment, the SharePoint on-premises and SharePoint Online analytics are completely silo’d from each other. To add insult to injury, Yammer has its own separate analytics, which is richer but locked down to OOTB formats available only to admins, and limited to the past 7 or 28 days. And extracting data from SharePoint Online is not easy; there is no client-side API available.
If you want to see what is going on with the usage of your hybrid system and use that insight to drive change or resolve problems, you will run into this, and it is an UGLY issue. Your alternatives are to find a third party product or embark on a daunting development project.
Ugly or Not, Hybrid is Here
Hybrid is here. It’s not easy, but it is often the best way to reap the benefits of cloud services and leverage new Microsoft capabilities. And in some scenarios it is the reality – really the only good solution.
There are many good things about Hybrid SharePoint, alongside the bad and the ugly. I have now seen enough success stories that I know that the good carries the day. And as I mentioned at the start of this article, we seem to have reached a tipping point with a surge of interest and activity around Hybrid.
For me personally, this has been both challenging and fun. I really enjoy working make sure that our customers have a smooth migration experience, and developing new capabilities to help them get the best of both worlds. I’ve learned a lot from working with them and in particular discovering where OOB hybrid search is just not enough and how to fix it.
As you’ve noticed, Microsoft’s “cloud-first/cloud-only” strategy has also meant a dramatic change in the way new capabilities are announced and released. While this new rhythm takes a lot of adjustment, it does provide a regular stream of improvements and fixes. The dramatic improvement in areas like directory synchronization and authentication for Hybrid SharePoint are a case in point.
The promise of Hybrid SharePoint – that you can enjoy the “best of both worlds” benefits of Cloud adoption sooner, and have lower risk and increased flexibility – has some real reality. Just watch out for the bad parts and the ugly parts.
What are you seeing with Hybrid SharePoint? Talk to us and share your experience and your feedback!
Jeff Fried
About the Author: Jeff Fried, BA Insight’s CTO, is focused on strategic applications of search technology. Prior to joining BA-Insight, Jeff was a VP of Advanced Solutions for FAST Search and Transfer, and with FAST’s acquisition by Microsoft, he served as core product manager for FAST Search for SharePoint and then technical product manager for all Microsoft enterprise search products. Jeff is a frequent speaker and writer in the industry; holds 15 patents; and has authored more than 50 technical papers. He is a co-author of three books: Professional SharePoint 2010 Development, Professional SharePoint 2013 Development and Professional Microsoft Search.
Follow Jeff on Twitter!
